Setting up Azure Blob Storage Log Delivery

RTLD may automatically deliver compressed log data to an Azure Blob Storage container by submitting HTTPS PUT requests to it. Each request creates a block blob within the container. This block blob contains a compressed JSON or CSV document that uniquely identifies a set of log data and describes one or more log entries.

Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

RTLD applies gzip compression to log data. Azure Blob Storage stores compressed log data as an object with a gz file extension.
Learn more.

Setting up log delivery to Azure Blob Storage requires:

In addition to the above requirements, you may specify an optional prefix that defines the location where log data will be uploaded within your container. Content will be uploaded to the location defined by this prefix as indicated by the following URL.

https://Storage Account.blob.core.windows.net/Container/Prefix

Key information:

To create a log delivery profile

  1. Create or identify an Azure storage account and a container to which log data will be posted.

    View Microsoft Azure documentation on how to create a storage account.

  2. Navigate to the Real-Time Log Delivery CDN | Rate Limiting | WAF page. From the main menu, navigate to More and then find Real-Time Log Delivery under Analytics. Select either CDN, WAF, or RL.

  3. Click Add Profile.
  4. From the Log Delivery Method option, select Azure Blob Storage.
  5. Set the Blob Container URL option to a URL that points to the container to which log data will be posted.

  6. Optional. Set the Prefix option to a value that defines a virtual log file storage location and/or a prefix that will be added to each log file added to your container.

    Learn more.

  7. From the Access Type option, select whether log data uploads will be authorized via a SAS token or an access key and then paste it in the field below it.

    If you plan on providing a SAS token, make sure that the token has permission to write to the blob/container. Additionally, it should start with sv= and it should not include a ?.

  8. From the Log Format option, select whether to format log data using our standard JSON format, as a JSON array, as JSON lines, or as a CSV (RTLD CDN only).

    Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

  9. From the Downsample the Logs option, determine whether all or downsampledReduces the amount of log data that will be delivered. For example, you may choose to only deliver 1% of your log data. log data will be delivered.

    • All Log Data: Verify that the Downsample the Logs option is disabled.
    • Downsampled Log Data: Downsample logs to 0.1%, 1%, 25%, 50%, or 75% of total log data by enabling the Downsample the Logs option and then selecting the desired rate from the Downsampling Rate option.

      Use this capability to reduce the amount of data that needs to be processed or stored by Azure Blob Storage.
      RTLD CDN Only: Downsampling log data also reduces usage charges for this service.

  10. Log delivery setup varies according to whether you are delivering log data for CDN traffic, threats identified by WAF, or rate limited requests.

  11. Set the Log Delivery Enabled option to the "on" position.

  12. Click Save.

Log File Naming Convention

RTLD CDN: On 12/8/2019, the log file naming convention was updated to include the profile ID for your Real-Time Log Delivery configuration.

The log data stored within an object is compressed using gzip. Each object follows this naming convention:

The JSON document contained within an object follows this naming convention:

Sample file name (RTLD CDN - JSON log format):

adn_0001_123_20220111_50550000F98AB95B_1.json

Sample file name (RTLD Rate Limiting - JSON log format):

rl_0001_123_20220111_50550000F98AB95B_1.json

Sample file name (RTLD WAF - JSON log format):

waf_0001_123_20220111_50550000F98AB95B_1.json

Each of the above file naming variables are described below.

Variable Description

Log Type

Represents the type of log data.

  • RTLD CDN: Identifies the delivery platform for which CDN activity was logged. Valid values are:

    • wpc: HTTP Large platform
    • wac: HTTP Small platform
    • adn: Application Delivery Network platform
  • RTLD Rate Limiting: This variable is always set to rl.
  • RTLD WAF: This variable is always set to waf.

AN

Represents your CDN account number (e.g., 0001). This account number may be viewed from the upper-right hand corner of the MCC.

Profile ID

Represents the system-defined ID for your Real-Time Log Delivery configuration.

You cannot currently view the system-defined ID assigned to your Real-Time Log Delivery configuration from within the MCC.

Date Stamp

Represents the date on which the log file was generated.

Syntax:

YYYYMMDD

Example:

20220110

Agent ID

Represents a unique ID that identifies the Real-Time Log Delivery software agent that generated the log file.

Sequence Number

Represents a sequential number that identifies the order in which the log file was generated by the software agent identified above.

Each software agent assigns a sequential number to the log files that it generates. A gap between log files generated on the same day by the same software agent indicates missing log data.
Learn more.

Key information:

  • This number starts at 0.
  • This number resets to 0 at the start of a new day (UTC).

File Extension

Represents the file extension for the log file. This file extension varies by log format.

  • JSON Log Format: json
  • JSON Array Log Format: json_array
  • JSON Lines Log Format: json_lines
More Information