All months

Changelog · 2025-01

January 2025

Crawler API improvements, a higher API rate limit of 120 requests a minute, and two new Terraform resources for managing crawlers and crawler schedules as code.

12 updates across 7 products shipped this month.

New 1

Back to top
New Terraform

Crawler and crawler schedule resources

The Terraform provider now includes quant_crawler and quant_crawler_schedule resources, so you can define crawlers and recurring crawls alongside your other Quant resources. A quant_crawler resource sets the crawler's domain, start URLs, excluded paths, request headers and browser mode. A quant_crawler_schedule resource manages the crawler schedules added to the v2 API in December, and each schedule takes a cron expression and an optional name.

Improved 9

Back to top
Improved API

Rate limits increased

The API rate limit has been raised from 30 to 120 requests per minute. This unblocks high-frequency use cases such as automated deployments and bulk operations via the CLI or CI/CD pipelines.

Improved CDN

Cache shielding defaults to off for new projects

New projects now have cache shielding turned off by default, and the shield region selector has a new none option. Previously, new projects defaulted to a shield region based on the project's region. Existing projects keep the shield region that is saved in their settings.

Improved Crawler

Include and exclude lists in the API

When you create or update a crawler through the API, you can now set an include list as well as an exclude list. Previously the API accepted only exclude. Each list accepts up to 20 regular expression patterns. When you set include, the crawler follows only the links that match at least one of its patterns.

Related documentation: Crawler: URL configuration

Improved Crawler

Iframe content and sitemap URLs

Crawls now also capture pages and files that your pages load in iframes, when they are on your domain or an allowed domain. Sitemap URLs are now crawled on the crawler's domain, even when the sitemap lists them under another host name, such as your public domain.

Improved Domains & DNS

Configuration guide always visible

The domain settings page now shows the DNS configuration instructions even when all domains are fully configured. Previously it was hidden once all domains were configured, which made it harder to find the A record and CNAME targets when you added a domain.

Improved Projects

Save now shows a loading state

Saving a project now shows a spinner and disables the submit button to prevent double-submission. The form was previously silent during save, which sometimes led to duplicate requests.

Improved Projects

Revisions disabled by default via API

The API create project endpoint now sets disable_revisions: true by default, avoiding unnecessary storage use for teams who don't use revision history. Previously projects created via the API had revision history enabled unless explicitly disabled.

Improved SDKs

disable_revisions support

The Go admin client now supports the disable_revisions field on project resources, consistent with the API and Terraform provider.

Improved Terraform

Project revisions setting and clearer errors

The quant_project resource has a new disable_revisions attribute, which defaults to true, the same as the API. When the provider cannot create a project, it now says whether the API token is invalid or lacks permission, or a project with that name already exists.

Set disable_revisions = false on quant_project resources that need revision history.

Fixed 2

Back to top
Fixed Crawler

Update API

The crawler update endpoint was partially broken — updates to start_urls, headers, and name were not being applied correctly. All fields now update as expected, and fields omitted from an update request are left unchanged.

Fixed Crawler

Include patterns and redirected files

An include pattern anchored at the start of a path, such as ^/news/, did not match any links. Include patterns now match each link's path and query string. A linked file URL that redirected to an HTML page was saved as a copy of that page. It is now saved as a redirect.