Show search and navigation.

 

Bot Manager Advanced

Applies To:
HTTP LargeThis platform has been optimized to cache and deliver static content (e.g., HTML, CSS, JavaScript, ISO, multimedia, and software downloads, etc.) over HTTP or HTTPS.
HTTP SmallLegacy. If you are currently serving traffic over this platform, then you may continue to do so. However, we recommend that you serve your traffic over our more robust HTTP Large platform.
ADNThe Application Delivery Network platform has been optimized to deliver dynamic content (e.g., login credentials, account information, etc.) over HTTP or HTTPS. Typically, user-specific and database-driven content are served over this platform.
WAFWeb Application Firewall is designed to secure site traffic against DDoS and application layer attacks. It also filters unwanted traffic by enforcing a custom delivery profile and whitelists, access list, and blacklists. Finally, it controls the flow of requests to your applications.
ClosedWhat are the differences between Bot Manager Standard and Advanced?

Bot Manager Standard is designed to mitigate basic bots through a browser challenge. Bot Manager Advanced has this capability, but it also automatically detects good bots and bots that are spoofing good bots. Additionally, you may define a custom traffic profile through which bots will be identified and mitigated.

Bot Manager Advanced adds an additional layer of security that is dedicated to bot detection and mitigation. It is designed to automatically detect good bots (e.g., search bots) and bad bots, including those that spoof good bots, by analyzing requests and behavior. You may even customize how bad bots are detected and mitigated by defining custom criteria that profiles a bad bot and the action that we will take for that traffic. Bot Manager Advanced is also able to mitigate basic bots by requiring a web browser to resolve a JavaScript challenge before our service will resolve traffic. Finally, it provides actionable near real-time data on detected bots through which you may fine-tune your configuration to reduce false positives.

Bot Manager Advanced is a powerful tool through which you may mitigate undesired bot traffic and prevent them from performing undesired or malicious activity, such as scraping your siteRefers to harvesting data from your site., cardingRefers to the process through which stolen credit cards are authorized., taking over accounts through credential stuffing, spamming your forms, launching DDoS attacks, and committing ad fraud.

How Does It Work?

Bot Manager Advanced inspects each request to determine whether the client:

  1. Matches a known good bot (e.g., search bot).
  2. Is spoofing a known good bot.

  3. Matches a rule. A rule defines the criteria that our service will use to identify a bad bot.

    You may identify bots using:

    • Information derived from the request, such as geolocation, IP address, and the URL path.

    • Our request and behavioral analysis that assigns a bot score to the request that defines our level of confidence that it is a bot.

      You may set actions based off of bot score thresholds.

      For example, you may redirect requests whose bot score is between 50 and 80% and block requests whose bot score is greater than 80%.

    • The JA3 fingerprint assigned to the request. A JA3 fingerprint identifies a client using key characteristics from a TLS request. This allows us to classify traffic as a specific bot across various IP addresses and ports.

Key information:

Actions

The type of actions that can be applied to bot traffic are:

Mode Description

Alert

Generates an alert.

Use this mode to track detected threats through the Bots dashboard without impacting production traffic.

Browser challenge

Sends a browser challenge to the client. The client must solve this challenge within a few seconds.

Response

The results of the above browser challenge determines what happens next.

  • Solved: If the client is able to solve the challenge, then our CDN serves the requested content. Additionally, a cookie will be added to the user's session. This cookie instructs our CDN to serve content to the user without requiring a browser challenge. Once the cookie expires, new requests for content protected by Bot Manager will once again require the client to solve a challenge.

    Define the duration for this cookie through the Valid for (in minutes) option.

  • Unsolved: If the client is unable to solve the challenge, then our CDN responds with a new browser challenge.

Key information:

  • Solving a challenge requires a JavaScript-enabled client. Users that have disabled JavaScript on their browsing session will be unable to access content protected by browser challenges.

  • We strongly recommend that you avoid applying browser challenges to machine-to-machine interactions.

    For example, applying browser challenges to API traffic will disrupt your API workflow.

  • The HTTP Status Code option determines the HTTP status code for the response provided to clients that are being served the browser challenge.

    Setting this option to certain status codes (e.g., 204) may prevent clients that successfully solve a browser challenge from properly displaying your site.

  • You may define a custom payload for the browser challenge by enabling the Custom Browser Challenge Page option and then setting the Browser Challenge Page Template option to the desired payload.

Block

Drops the request and the client will receive a 403 Forbidden response.

Custom Response

Returns a custom response.

  • Response body: Define the payload that will be delivered to the client.

    This option supports the use of event variables to customize the response.

    Sample payload for a HTML file:

    <!DOCTYPE html><html>

    <head><title>Page Not Found</title></head>

    <body>Page not found.</body>

    </html>

  • HTTP status code: Defines the HTTP status code that will be sent to the client.

  • Custom response headers: Defines one or more response headers that will be sent to the client. Define each custom response header on a separate line.

    Syntax:

    Name:Value

    Example:

    MyCustomHeader: True

    This option supports the use of event variables to customize the response.

    All characters, including spaces, defined before or after the colon will be treated as a part of the specified header name or value, respectively.

Redirect

Redirects requests to the specified URL.

Key information:

  • The HTTP status code for this response will be a 302 Found.

  • Set the URL option to the full URL to which requests will be redirected.

    Example:

    http://cdn.mydomain.com/marketing/busy.html

Bot Manager Configuration

Each rule within a Bot Manager configuration identifies bot traffic. Each rule contains:

A Bot Manager configuration may contain up to 10 rules.

Custom Bot Detection

A request must satisfy at least one rule before WAF will consider it bot traffic. There are two types of rules, which are:

ClosedView examples

Example #1

This example assumes that your Bot Manager configuration contains the following two rules:

Rule

Type

 Description

1

Custom matches

This rule contains a single condition.

2

Custom matches

This rule contains two conditions.

Assuming the above configuration, WAF identifies bot traffic whenever either of the following conditions are met:

  • A match is found for the variable defined in the first rule's condition.

  • A match is found for the variables defined in both of the second rule's conditions.

Example #2

This example assumes that your Bot Manager configuration contains the following two rules:

Rule

Type

 Description

1

Custom matches

This rule contains two conditions.

2

Edgecast Reputation DB

This rule is satisfied when the client's IP address matches an IP address within our reputation database.

Assuming the above configuration, WAF identifies bot traffic whenever either of the following conditions are met:

  • A match is found for the variables defined in both of the first rule's conditions.
  • The client's IP address matches an IP address within our reputation database.

Conditions

A condition determines how requests will be identified through variables, operators, match values, and negative matching.

Variables

A variable identifies the request element that WAF will analyze. We support the following request elements:

All variables support the ability to match on the number of times that a request element is found within the request. Set up a variable to match on the number of instances instead of inspecting the element for a specific value or regular expression pattern by marking the Count option.

You may define zero or more keys when setting up variables that match on key-value pairs. WAF must find at least one of the specified keys in the request before that variable will be satisfied. For example, if you set up a request header variable to match for Authorization and Content-Type, then requests that contain either or both of those headers will satisfy that variable.

Operators

An operator determines how WAF will compare a match value against the request element identified by a variable.

Match Value

WAF uses a match value to identify threats.

Example:

This example assumes the following configuration:

Variable: Request header = Authentication

Match value: 1

We will now examine how the Count option affects comparisons for this configuration.

The type of comparison that will be performed is determined by the Operator option.

Exceptions

You may exempt traffic from bot detection by URL, user agent, JA3 fingerprint, and cookie.

Key information:

Bot Manager Configuration Administration

You may create, modify, and delete Bot Manager configurations.

Key information:

To create a Bot Manager configuration

  1. Navigate to the Bot Manager page. ClosedHow?From the main menu, navigate to More | Security | WAF Tier | Security Rule Manager | Bot Manager.

  2. Click Add Bot Rule.
  3. In the Name option, type the unique name by which this Bot Manager configuration will be identified. This name should be sufficiently descriptive to identify it when setting up a Security Application Manager configuration.

  4. Optional. Set up a browser challenge, custom response, or redirect that can be applied to known bots, spoofed bots, and bots detected through rules.

    Unlike other actions, alert and block actions do not require configuration before they can be applied to bot traffic.

    1. From the Actions section, select the desired action:

      • Browser challenge: Perform the following steps:

        1. From the Valid for (in minutes) option, type the number of minutes for which our CDN will serve content to a client that solves a browser challenge without requiring an additional browser challenge to be solved. Specify a value between 1 and 1,440 minutes.

        2. From the HTTP status code option, determine the HTTP status code for the response provided to clients that are being served the browser challenge.

          Setting this option to certain status codes (e.g., 204) may prevent clients that successfully solve a browser challenge from properly displaying your site.

        3. Serve a custom browser challenge by enabling the Custom Browser Challenge Page option and then setting the Browser Challenge Page Template option to the desired payload.

      • Custom response: Perform the following steps:

        1. From the Response body option, specify the body of the response that will be sent to clients.
        2. From the HTTP status code option, determine the HTTP status code for the response that will be sent to clients.

        3. From the Custom response headers option, define each desired custom response header on a separate line.

          Example:

          MyCustomHeader: True
      • Redirect: Set the URL option to the full URL to which requests will be redirected.
    2. Repeat the previous step as needed.

  5. Perform the following steps to automatically detect known bots:

    1. From the Known bots tab, click + Add known bot.

    2. From the Bot Token option, select whether to apply an action to all known bots (all), a specific bot, or to 200+ bots (other):

      Choose other to apply an action to 200+ known good bots. This option excludes the known bots that are listed within the Bot Token option.

    3. From the Action type option, select the action that will be applied to the known bot(s) selected in the previous step.

    4. Repeat the above steps as needed.

  6. The Spoofed Bots section in the Known bots tab determines how to handle traffic spoofing the known bots selected in the previous step. From the Action type option, select the desired action.

    The Spoofed Bots section does not apply to the 200+ known bots defined within the other category.

  7. Create rules for identifying bots from the Rules tab.

    1. In the Bot ruleset name option, type a unique name.
    2. Click + Add Rule.

    3. In the Rule type option, select the type of rule that will be created.

      • Custom Matches: This type of rule is satisfied when a match is found for each of its conditions.

        1. In the Name option, type a name that identifies the purpose of this rule.
        2. In the Rule ID option, specify a number between 77,000,000 and 77,999,999.
        3. In the Rule message option, type a brief description for this rule.

        4. A custom matches rule automatically includes a default condition. Modify this condition to determine how WAF will identify requests. From the condition's Variable option, select the request element through which WAF will identify requests.

          Learn more about variables.

        5. Certain variables (e.g., request cookies and request header) match on name and value. If you have selected this type of variable, then perform the following steps:

          1. Click + Add Match.

          2. From the Name option, type the desired name.

            For example, match for requests that contain an Authorization header by setting this option to Authorization.

          3. Optional. Mark the Negative Match option to match for requests that do not contain a matching value for the name defined in the previous step.
          4. If you specified a regular expression in the Name option, then you should mark the Regex Match option.
          5. Optional. Add another match through which this variable can be satisfied by repeating steps a - d.

        6. Optional. Mark the Count option to match by the number of instances that a match is found instead of by inspecting that request element.

          Learn more.

        7. From the Operator option, select an operator that determines how WAF will compare the match value to the request element identified by the above variable.

          Learn more.

        8. In the Match value option, type the value that will be compared against the request element identified by the above variable.
        9. Optional. Mark the Negative Match option to match for requests that do not contain a matching value for the value defined in step vi.
        10. Optional. Click + Add Condition to add another condition that must be met prior to request identification. Repeat steps i - ix for this new condition.
      • Edgecast Reputation DB: This type of rule is satisfied when the client's IP address matches an IP address within our reputation database. Proceed to the next step.
  8. Optional. Click + Add Rule to add another rule through which WAF may identify bots. Repeat step 7.3.

  9. Optional. Identify traffic that will bypass bot detection.

    1. Click the Exceptions tab.

    2. Add the desired URL(s), user agent(s), JA3 fingerprint(s), and cookie(s) as exception(s).

      Place each entry on a separate line.

  10. Click Save.

To modify a Bot Manager configuration

  1. Navigate to the Bot Manager page. ClosedHow?From the main menu, navigate to More | Security | WAF Tier | Security Rule Manager | Bot Manager.

  2. Click on the desired Bot Manager configuration.

  3. Make the desired changes.

    Key tasks:

    • Add, modify, or delete an action.
    • Set or modify the action that will be taken for all known bots, specific known bots, and spoofed bots from the Known Bots tab.
    • Update your exceptions to bot detection by adding, modifying, or deleting entries from the Exceptions tab.
    • Change the type of rule from the Rule type option.

    • Custom matches only

      Delete variables and matches within a variable by clicking .

    • Custom matches only

      Delete a condition by clicking Delete Condition.

      A rule must have at least one condition. Therefore, you cannot delete the root condition.

    • Delete a rule by clicking Delete Rule and then clicking Confirm.
  4. Click Save.

To delete a Bot Manager configuration

  1. Navigate to the Bot Manager page. ClosedHow?From the main menu, navigate to More | Security | WAF Tier | Security Rule Manager | Bot Manager.

  2. Click on the desired Bot Manager configuration.
  3. Click Delete Bot Manager Config.
  4. Type DELETE.
  5. Click Delete.