All spotlights

Product spotlight · July 2024

Block AI crawlers and serve your own WAF error pages

The Quant WAF can now block the crawlers that AI platforms use to collect training data, and you can replace its block and rate-limit pages with your own.

Updated Web application firewall For government and enterprise

What changed in July 2024

The July 2024 changelog has the full details.

  • Block AI crawlers and bots

    You can now block the crawlers and bots that large AI platforms use to collect training data. Turn on the option in a proxy rule's WAF settings, and the WAF blocks known AI crawlers at the edge. Quant maintains the list centrally, and you can filter the WAF logs to show these blocks.

  • Custom block and rate-limit pages

    You can now replace the default pages that the WAF serves for block and rate-limit responses with your own branded content. Publish the block page at the _quant_waf_403 route and the rate-limit page at the _quant_waf_429 route.

Also from April 2024

The April 2024 changelog has the full details.

  • Rate limiting in log filters

    The type filter on WAF logs now includes WAF hit rate limiting. You can narrow the WAF log view to only the requests that were rate-limited for repeatedly triggering WAF rules.

Who it is for

  • Public sector web teams

    Block the known crawlers that AI platforms use to collect training data from the content of your public sites.

  • Security teams

    Filter the WAF logs to see AI crawler blocks, or the requests rate-limited for repeatedly triggering WAF rules.

  • Site owners

    Show visitors a branded page that matches the rest of your site when the WAF blocks them or rate-limits their requests.

How it works

This month adds two WAF changes: an option that blocks AI crawlers and bots, and your own pages for block and rate-limit responses.

Blocking AI crawlers

Large AI platforms use crawlers and bots to collect training data from websites. If you do not want your content used in this way, open a proxy rule’s WAF settings. In the block dictionaries section, turn on “Block AI crawlers and bots”. The WAF then blocks known AI crawlers at the edge.

Quant maintains the list of AI crawlers centrally, so you do not need to keep your own list. To see the AI crawler requests that the WAF blocked, filter the WAF logs to show these blocks.

Your own error pages

When the WAF blocks a request or rate-limits a client, it serves a default page. You can replace each of these pages with your own branded content. Publish each page at its route:

_quant_waf_403   block page
_quant_waf_429   rate-limit page

You publish these pages in the same way as custom static error pages. The custom error pages guide has the steps.

Rate-limited requests in the logs

WAF hit rate limiting applies to clients that repeatedly trigger WAF rules. Since April 2024, the type filter on WAF logs includes it. Use the filter to narrow the log view to only those requests.

Get started

  1. Open the WAF settings

    Open the WAF settings of a proxy rule in your project.

  2. Block AI crawlers

    Turn on "Block AI crawlers and bots" in the block dictionaries section.

  3. Publish a block page

    Publish your own block page at the _quant_waf_403 route.

  4. Publish a rate-limit page

    Publish your own rate-limit page at the _quant_waf_429 route.

  5. Check the WAF logs

    Filter the WAF logs to see the AI crawler blocks.

Related: Explore Web application firewall · Documentation · July 2024 changelog

Ready to try it on Quant?

Use the web application firewall on your own project, or talk to our team first.