What changed in August 2025
The August 2025 changelog has the full details.
Remote URL lists
A crawl can now start from a remote URL list. You give the crawler the URL of a hosted list of paths. The crawler fetches the list and adds those paths to the crawl, in addition to any start URLs that you configured.
Webhook authentication header
Crawler webhooks now support an authentication header, in addition to the webhook URL and custom variables. The endpoint that you trigger when a crawl completes can use this header to verify the request.
Webhooks in the crawler API
You can now configure crawler webhooks through the crawler API, as well as in the dashboard.
One purge after each crawl
The crawler now runs one bulk cache purge at the end of a crawl. Before, it purged each item separately.
Terraform crawler resource
The
updated_atattribute of thequant_crawlerresource now shows the latest value from the API after each change.
Who it is for
Website teams
Give a crawl a hosted list of paths, so the crawler adds all of those paths in addition to your configured start URLs.
Integration developers
Check the authentication header in the endpoint that a completed crawl triggers, and use it to verify the request.
Platform teams
Configure crawler webhooks through the crawler API, and manage crawlers in code with the
quant_crawlerresource.
How it works
The site crawler visits the pages of a website and stores a static copy of them in your Quant project. A crawl starts from the start URLs in its configuration and follows the links that it finds on those pages.
Remote URL lists. A crawl can now also start from a list of paths that you host. You give the crawler the URL of the list. The crawler fetches the list and adds its paths to the crawl. Your configured start URLs still apply, so the crawl uses both. This helps when a page is on your list but no crawled page links to it.
Webhook authentication. The October 2024 changelog covered the crawl start event and custom webhook variables. A crawler webhook now has three parts:
- the webhook URL, which receives the notification
- custom variables, as key-value pairs in the payload
- an authentication header
When a crawl completes, the crawler calls your endpoint and sends the header with the request. Your endpoint can check the header value to verify the request before it acts on it.
Webhooks in the crawler API. You can now configure crawler webhooks through the crawler API, as well as in the dashboard.
One purge per crawl. The crawler now runs one bulk cache purge when the crawl ends, instead of a purge for each item that it stores.
Get started
Open the crawler
In the dashboard, select your project and open Crawler > Configs.
Set up a webhook
Enter your webhook URL and any custom variables.
Add the header
Enter an authentication header for the webhook.
Verify the request
Check the header in the endpoint that a completed crawl triggers.
Related: Explore Site crawler · Documentation · August 2025 changelog
Ready to try it on Quant?
Use the site crawler on your own project, or read the docs first.