Setting up Google Cloud Storage Log Delivery

RTLD may automatically deliver compressed log data to a Google Cloud Storage bucket by submitting HTTPS PUT requests to it. Each request adds an object to a Cloud Storage bucket. This object contains a compressed JSON or CSV document that uniquely identifies a set of log data and describes one or more log entries.

Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

RTLD applies gzip compression to log data. Google Cloud Storage stores compressed log data as an object with a gz file extension.
Learn more.

To create a log delivery profile

  1. Create or identify a Google Cloud Storage bucket to which log data will be posted.

    Key information:

    • The recommended configuration is to set the Access control option to Uniform.
    • Set the Encryption option to Google-managed encryption key.

    View Google Cloud Storage documentation on how to create a bucket.

  2. Add the following user to the bucket and assign it the Storage Object Creator role:

    vdms-partner-gcs-transfer@maw-partner-gcs.iam.gserviceaccount.com

    View Google Cloud Storage documentation on how to set up an IAM policy for a bucket.

  3. Optional. Set up Google Cloud to process the log data that will be posted to it.

    Example:

    Load logs into BigQuery and then leverage BigQuery functionality through the BigQuery Browser Tool.

  4. Navigate to the Real-Time Log Delivery CDN | Rate Limiting | WAF page. From the main menu, navigate to More and then find Real-Time Log Delivery under Analytics. Select either CDN, WAF, or RL.

  5. Click Add Profile.
  6. From the Log Delivery Method option, select Google Cloud Storage.
  7. Set the Bucket option to the name of the Google Cloud Storage bucket to which log data will be posted.
  8. Optional. Set the Prefix option to the desired prefix that defines a virtual log file storage location and/or a prefix that will be added to each object added to your bucket.

  9. From the Log Format option, select whether to format log data using our standard JSON format, as a JSON array, as JSON lines, or as a CSV (RTLD CDN only).

    Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

  10. From the Downsample the Logs option, determine whether all or downsampledReduces the amount of log data that will be delivered. For example, you may choose to only deliver 1% of your log data. log data will be delivered.

    • All Log Data: Verify that the Downsample the Logs option is disabled.
    • Downsampled Log Data: Downsample logs to 0.1%, 1%, 25%, 50%, or 75% of total log data by enabling the Downsample the Logs option and then selecting the desired rate from the Downsampling Rate option.

      Use this capability to reduce the amount of data that needs to be processed or stored by Google Cloud Storage.
      RTLD CDN Only: Downsampling log data also reduces usage charges for this service.

  11. Log delivery setup varies according to whether you are delivering log data for CDN traffic, threats identified by WAF, or rate limited requests.

  12. Set the Log Delivery Enabled option to the "on" position.

  13. Click Save.

Log File Naming Convention

The log data stored within an object is compressed using gzip. Each object follows this naming convention:

The JSON document contained within an object follows this naming convention:

Sample file name (RTLD CDN - JSON log format):

adn_0001_123_20230111_50550000F98AB95B_1.json

Sample file name (RTLD Rate Limiting - JSON log format):

rl_0001_123_20230111_50550000F98AB95B_1.json

Sample file name (RTLD WAF - JSON log format):

waf_0001_123_20230111_50550000F98AB95B_1.json

Each of the above file naming variables are described below.

Variable Description

Log Type

Represents the type of log data.

  • RTLD CDN: Identifies the delivery platform for which CDN activity was logged. Valid values are:

    • wpc: HTTP Large platform
    • wac: HTTP Small platform
    • adn: Application Delivery Network platform
  • RTLD Rate Limiting: This variable is always set to rl.
  • RTLD WAF: This variable is always set to waf.

AN

Represents your CDN account number (e.g., 0001). This account number may be viewed from the upper-right hand corner of the MCC.

Profile ID

Represents the system-defined ID for your Real-Time Log Delivery configuration.

You cannot currently view the system-defined ID assigned to your Real-Time Log Delivery configuration from within the MCC.

Date Stamp

Represents the date on which the log file was generated.

Syntax:

YYYYMMDD

Example:

20230110

Agent ID

Represents a unique ID that identifies the Real-Time Log Delivery software agent that generated the log file.

Sequence Number

Represents a sequential number that identifies the order in which the log file was generated by the software agent identified above.

Each software agent assigns a sequential number to the log files that it generates. A gap between log files generated on the same day by the same software agent indicates missing log data.
Learn more.

Key information:

  • This number starts at 0.
  • This number resets to 0 at the start of a new day (UTC).

File Extension

Represents the file extension for the log file. This file extension varies by log format.

  • JSON Log Format: json
  • JSON Array Log Format: json_array
  • JSON Lines Log Format: json_lines
More Information