What changed in July 2024
The July 2024 changelog has the full details.
Block AI crawlers and bots
You can now block the crawlers and bots that large AI platforms use to collect training data. Turn on the option in a proxy rule's WAF settings, and the WAF blocks known AI crawlers at the edge. Quant maintains the list centrally, and you can filter the WAF logs to show these blocks.
Custom block and rate-limit pages
You can now replace the default pages that the WAF serves for block and rate-limit responses with your own branded content. Publish the block page at the
_quant_waf_403route and the rate-limit page at the_quant_waf_429route.
Also from April 2024
The April 2024 changelog has the full details.
Rate limiting in log filters
The type filter on WAF logs now includes WAF hit rate limiting. You can narrow the WAF log view to only the requests that were rate-limited for repeatedly triggering WAF rules.
Who it is for
Public sector web teams
Block the known crawlers that AI platforms use to collect training data from the content of your public sites.
Security teams
Filter the WAF logs to see AI crawler blocks, or the requests rate-limited for repeatedly triggering WAF rules.
Site owners
Show visitors a branded page that matches the rest of your site when the WAF blocks them or rate-limits their requests.
How it works
This month adds two WAF changes: an option that blocks AI crawlers and bots, and your own pages for block and rate-limit responses.
Blocking AI crawlers
Large AI platforms use crawlers and bots to collect training data from websites. If you do not want your content used in this way, open a proxy rule’s WAF settings. In the block dictionaries section, turn on “Block AI crawlers and bots”. The WAF then blocks known AI crawlers at the edge.
Quant maintains the list of AI crawlers centrally, so you do not need to keep your own list. To see the AI crawler requests that the WAF blocked, filter the WAF logs to show these blocks.
Your own error pages
When the WAF blocks a request or rate-limits a client, it serves a default page. You can replace each of these pages with your own branded content. Publish each page at its route:
_quant_waf_403 block page
_quant_waf_429 rate-limit pageYou publish these pages in the same way as custom static error pages. The custom error pages guide has the steps.
Rate-limited requests in the logs
WAF hit rate limiting applies to clients that repeatedly trigger WAF rules. Since April 2024, the type filter on WAF logs includes it. Use the filter to narrow the log view to only those requests.
Get started
Open the WAF settings
Open the WAF settings of a proxy rule in your project.
Block AI crawlers
Turn on "Block AI crawlers and bots" in the block dictionaries section.
Publish a block page
Publish your own block page at the
_quant_waf_403route.Publish a rate-limit page
Publish your own rate-limit page at the
_quant_waf_429route.Check the WAF logs
Filter the WAF logs to see the AI crawler blocks.
Related: Explore Web application firewall · Documentation · July 2024 changelog
Ready to try it on Quant?
Use the web application firewall on your own project, or talk to our team first.