The format in which data generated for CDN activity over the HTTP protocol is stored in a raw log file is determined by the settings defined in the Log Format Settings section of the Raw Log Settings page. These log file formatting settings allow you to:
- Define the base log file format
- Define the date/time format
- Determine whether a custom field will be included
- Determine whether the content access point will be logged
Typically, data mining tools require a consistent log format when analyzing data over a period of time. If you plan on using a data mining application on your log data, then it is key to keep changes to your log file to a minimum. Please contact your CDN account manager before making any changes.
Base Log File Format
The base log file format determines the basic format that will be used to record HTTP activity in a raw log file. Once you have selected a base log file format, you can adjust other log file format settings (e.g., date/time format) to achieve your optimal log format. When choosing a base log file format, you will need to choose between the following two log file formats:
- Default format: The default log file format is similar to an extended W3C log file format.
- Combined format: A combined log file format is similar to the default log file format used by Apache web servers. The following list of fields indicates the type of data and the order in which it is recorded in a raw log file:
- Client IP Address
- Client Identification
- Client User Name - HTTP Authentication
- Date/Time
- Request URL
- [HTTP] Status Code
- File Size
- Referrer
- User Agent
- Custom Log Field 1
Key information about the combined log file format:
- Unlike the default log file format, the combined format does not provide a list of field headers in each raw log file.
- By default, the Client Identification and the Client User Name – HTTP Authentication fields report a dash (i.e., -) for each request.
-
Custom Log Field 1 will only be reported if custom logging is turned on when the CDN activity being reported takes place.
Date/Time Format
A CDN server records the date and time at which it processed the request for an asset in a log field called "timestamp." The format in which data is recorded in this field can be customized in one of the following ways:
- Unix time (Unix Epoch): Date and time can be recorded in the timestamp field as the number of seconds since Unix time (a.k.a. POSIX time or Unix epoch). Unix time starts on 1970-01-01 at 00:00:00 GMT. For example, if the timestamp field reports "1294401600" then the request was processed on 1/07/2011 at 12:00:00 GMT. This is the default format for the timestamp field.
- Custom date/time (Default format): If the base log file format is set to Default format, then selecting the Custom date/time format option will record date and time in the timestamp field using the following format: YYYY-MM-DD hh:mm:ss (e.g., 2024-01-07 12:00:00). Each date/time variable is defined below.
- YYYY: Indicates a year in the Gregorian calendar using a four digit number (e.g., 2024).
- MM: Indicates a month of the year between 01 (January) and 12 (December).
- DD: Indicates a day of the month between 01 and 31.
- hh: Indicates an hour of the day (GMT) between 00 and 24.
- mm: Indicates minutes between 00 and 59.
- ss: Indicates seconds between 00 and 59.
- Custom date/time (Combined format): If the base log file format is set to Combined format, then selecting the Custom date/time format option will record date and time in the timestamp field using the following format: DD/MMM/YYYY:hh:mm:ss +hhhh (e.g., 07/Jan/2024:12:00:00 +0000). Each date/time variable is defined below.
- DD: Indicates a day of the month between 01 and 31.
- MMM: Indicates a three letter abbreviation for a month of the year (e.g., Jan).
- YYYY: Indicates a year in the Gregorian calendar using a four digit number (e.g., 2024).
- hh: Indicates an hour of the day between 00 and 24.
- mm: Indicates minutes between 00 and 59.
- ss: Indicates seconds between 00 and 59.
- +hhhh: Indicates the hour differential between the time reported and GMT.
Custom Field
By default, a custom field called "x-ec_custom-1" is included in your raw log files. This field will report "-" unless Rules Engine has been configured to log request/response header data. This type of configuration requires the use of the Custom Log Field 1 feature.
Changing the name of the custom field will not affect whether HTTP request and/or response headers are logged.
Raw log files that use the combined log format will not indicate the name of the custom log field.
When configuring the Custom Log Field 1 feature, specify the name of each request and response header that will be logged in that field. A header can be specified using the following syntax:
| Header Type | Syntax | Example |
|---|---|---|
|
Request |
%{requestheader}i |
%{Accept-Encoding}i |
|
Response |
%{responseheader}o |
%{Content-Type}o |
Key information about logging header data:
- The Add this custom field to the log file option determines whether the custom field will report a dash or header information. Clearing this option will prevent Rules Engine from logging HTTP request and/or response headers in your log file.
- When specifying a header, make sure to add an indicator (i.e., i or o) to indicate whether it is a request (inbound) or response (outbound) header (e.g., %{Cookie}i or %{Content-Range}o).
-
It is recommended to use a separator when specifying multiple headers.
Example:
In the following sample value, an abbreviation for each header is used to delimit them.
AE: %{Accept-Encoding}i CT: %{Content-Type}o - Ensure that header data is reported for all transactions for a particular platform by adding the Custom Log Field 1 feature to a rule that is set to match "Always."
To determine whether header information will be logged
-
Navigate to the Raw Log Settings page.
How? -
Perform one of the following steps:
-
To log header information:
-
Make sure that the Add this custom field to the log file option is marked. If desired, change the custom field name. Click Update if this step required a change to this setting.
- Navigate to the Rules Engine page corresponding to the platform for which header information will be logged.
-
If a policy has not been deployed to the Production environment or does not contain the Custom Log Field 1 feature, then please create one that does.
A copy of the policy currently deployed to the Production environment may be created. Modify the resulting draft to include the Custom Log Field 1 feature for the desired types of requests (e.g., Always). After which, convert the draft to a policy and then deploy it to the Production environment.
- Repeat the previous step for each desired platform.
Append the appropriate identifier (i.e., i or o) to the header name to indicate whether you would like to log requests (inbound) or responses (outbound).
-
-
To exclude header data from the custom log field:
- Make sure that the Add this custom field to the log file option is cleared. Click Update if this required a change to this setting.
- Navigate to the Rules Engine page.
- Create a copy of the policy currently deployed to the Production environment.
- Remove all instances of the Custom Log Field 1 feature from the resulting draft.
- Convert the draft to a policy and then deploy it to the Production environment.
-
Content Access Point Logging
A content access point is a relative path that starts directly after the CDN hostname (e.g., /800001/MyOrigin or /000001). By default, this information is reported in the cs-uri-stem field in the log file. The cs-uri-stem field identifies the asset that was requested by the client. The Remove the content access point from the URL option determines whether this information is reported in that field.
Sample cs-uri-stem field values are provided below.
- Remove the content access point from the URL option (Enabled):http://
- Remove the content access point from the URL option (Disabled):http://
Edgecast CDN