What changed in October 2024
The October 2024 changelog has the full details.
Crawl start webhook event
Crawler webhooks can now notify you when a crawl starts. Your downstream systems learn about a crawl as soon as it begins.
Custom webhook variables
You can add your own custom variables, as key-value pairs, to webhook payloads. Downstream systems then receive the context they need without extra lookups.
Schedules in plain English
Scheduled crawl configurations now show their timing as a readable description alongside the cron expression. This makes it easier to confirm when a crawl will run.
Also from September 2024
The September 2024 changelog has the full details.
Headless mode and defaults
When you created a crawler, the headless mode setting was not applied correctly, and some default crawler settings were inconsistent. These and other issues with creating and configuring crawlers are now resolved.
Who it is for
Website teams
Keep a static copy of a dynamic site in Quant, and let other systems know as soon as each crawl of the site starts.
Integration developers
Add your own key-value variables to webhook payloads, so the systems that receive them need no extra lookups.
Site administrators
Read each crawl schedule as a plain description next to its cron expression, and confirm when a crawl will run.
How it works
The site crawler visits the pages of a website and stores a static copy of them in your Quant project. You can then serve a dynamic site, or a site from a CMS without a native integration, as static content. You can run a crawl when you need it, or set a schedule so that crawls run automatically.
Crawler webhooks tell other systems about crawl events. In October 2024, crawler webhooks changed in two ways:
- Crawl start event. The crawler can now send a webhook when a crawl starts. Your downstream systems then know about a crawl as soon as it begins.
- Custom variables. You can add your own custom variables, as key-value pairs, to the webhook payload. The systems that receive the webhook then get the context they need, without extra lookups.
A crawl schedule uses a cron expression to set when the crawl runs. A cron expression is short, but it can be difficult to read. Each scheduled crawl configuration now also shows its timing as a readable description, next to the cron expression. You can check the description to confirm that a crawl will run at the time you expect.
In September 2024, a fix made the headless mode setting apply correctly when you create a crawler. Use headless mode for sites that need JavaScript to render their pages. The same fix made some default crawler settings consistent and resolved other issues with creating and configuring crawlers.
Get started
Open the crawler
In the dashboard, select your project and open Crawler > Configs.
Add a crawler
Choose Add, enter the domain, and turn on Headless browser for JavaScript sites.
Set up a webhook
Add your webhook URL and any custom variables as key-value pairs.
Schedule your crawls
In Crawler > Schedules, choose Add, pick the configuration and set the timing.
Check the timing
Read the description next to the cron expression to confirm when it runs.
Related: Explore Site crawler · Documentation · October 2024 changelog
Ready to try it on Quant?
Use the site crawler on your own project, or talk to our team first.