All months

Changelog · 2024-12

December 2024

A rebuilt site crawler with new control options, a CDN metrics v2 API, crawler scheduling through the API, and key-value stores you can place in a region.

9 updates across 5 products shipped this month.

New 3

Back to top
New CDN

Metrics v2 API

A new metrics API is available under /v2/organizations/{org}/projects/{project}/metrics. You can request traffic, cache, response status, and origin status metrics at minute, day, month, or total granularity over a chosen time range, with results broken down per domain. A companion metrics/available endpoint lists the metrics and granularities you can query.

New Crawler

Rebuilt site crawler

The site crawler has been rebuilt with broader content coverage and finer control over what it captures. New options include sitemap.xml discovery, single-URL crawls, allowed-domain and domain-merging controls, and include and exclude regex patterns. You can also match file extensions with wildcards and rewrite links to relative paths. Browser-based crawling now scrolls pages to load lazy-loaded content, and it detects non-HTML responses and stores them as files. It also handles redirects and file downloads more reliably.

Related documentation: Crawler: Configuration options

New Crawler

Scheduling via API

You can now create, list, update and delete crawler schedules through the v2 API. Each schedule takes a cron expression and an optional name, so you can manage recurring crawls programmatically, not only in the dashboard. The Go admin client also manages schedules.

Related documentation: Crawler: Scheduling

Improved 5

Back to top
Improved CDN

Longer failover origin timeout

The maximum time-to-first-byte for a failover origin has been raised from 15 seconds to 60 seconds, giving slower failover origins more time to respond before a request is abandoned.

Improved CDN

URL purges cover every letter case

When you purge a path, cached copies of that path are now cleared whatever letter case visitors used to request it. For example, a purge of /about also clears responses cached for /About or /ABOUT, so mixed-case links no longer keep serving stale content after a purge.

Improved KV stores

Regional store placement

Key-value stores are now created in their project's region (Australia, United States or Europe), so stored data stays in that region.

Improved SDKs

Custom endpoint and file headers in the TypeScript content client

Version 2.0.6 of @quantcdn/quant-client, the TypeScript client for the content API, adds an endpoint configuration option. The option sets the base URL for content and search requests, and defaults to https://api.quantcdn.io/v1. In version 2.0.7, project.file() also accepts headers: a JSON string of response headers to serve with the file, such as a content-type for a file without an extension.

Improved Search

Multi-value field indexing

When extracting content for search, pipe-separated values in a single field are now split, so multi-value attributes index correctly.

Fixed 1

Back to top
Fixed Crawler

Runs now report errors

Crawler runs that failed to return a status were previously left without a final state. They are now marked as errored and closed off correctly, so a failed crawl no longer appears to be running indefinitely.