# Common Crawl Web Archive Index API

> Common Crawl: look up any URL or domain in the open web crawl archive, with WARC file locations

- Vendor: `commoncrawlwebarchiveindexapi` · version 1.0 · category Search · tags common crawl api, web archive api, url index api, cdx api, web crawl data api, warc lookup api
- Datasource: custom HTTP routes
- Marketplace page: https://apiclay.com/api/commoncrawlwebarchiveindexapi
- This document: https://apiclay.com/docs/commoncrawlwebarchiveindexapi.md
- OpenAPI 3.1: https://apiclay.com/api/commoncrawlwebarchiveindexapi/openapi.json
- MCP: connect https://api.apiclay.com/v1/mcp (see https://apiclay.com/llms.txt), then `apiclay_describe_api` with vendor `commoncrawlwebarchiveindexapi`
- Updated: 2026-11-18

## About

## Common Crawl Web Archive Index API

The CDX index server of Common Crawl, the non-profit open web crawl used to train and test search and AI systems: the list of monthly crawls since 2008, and for any crawl every capture of a URL, a URL prefix or a whole domain, with timestamp, status, MIME type, detected language, content digest and the WARC file, offset and length to fetch the page from the public archive.

Data comes from **commoncrawl.org** (https://commoncrawl.org). Every route calls the source's own public API and returns it unchanged.

Typical flow: `POST /crawls` for the latest crawl `id`, then `POST /lookup?url=example.com&amp;matchType=domain&amp;limit=50` with the body `{"crawl": "CC-MAIN-2026-43"}`.

### Routes

#### POST `/crawls` - Crawls
No parameters.

Returns: Array of crawls, newest first: `id` (e.g. `CC-MAIN-2026-43`), `name`, `from`, `to`, `cdx-api`, `timegate`.

#### POST `/lookup` - URL lookup
| Param | In | Type | Required | Description |
|---|---|---|---|---|
| crawl | body | string | yes | Crawl `id` from `/crawls`, e.g. `CC-MAIN-2026-43`. |
| url | query | string | yes | URL or domain, e.g. `example.com` or `example.com/blog/*`. |
| matchType | query | string | no | `exact`, `prefix`, `host` or `domain`. |
| limit | query | number | no | Captures to return, e.g. `50`. |
| filter | query | string | no | Field filter, e.g. `status:200` or `mime:text/html`. |
| from | query | string | no | Earliest timestamp, `YYYYMMDDhhmmss` or a prefix. |
| to | query | string | no | Latest timestamp. |

Returns: Newline-delimited JSON, one capture per line: `urlkey`, `timestamp`, `url`, `mime`, `mime-detected`, `status`, `digest`, `length`, `offset`, `filename` (WARC path under https://data.commoncrawl.org/), `languages`, `encoding`.

### Caveats

- Index lookups can take several seconds; a whole-domain lookup on a big site can return many MB, so set `limit`.
- Common Crawl's index server is busy at times and answers 504; retry after a few seconds.
- Upstream responses can change over time. Verify fields before relying on them.
- Not affiliated with or endorsed by commoncrawl.org.

## Pricing

- Pay as you go: 0.01 credits/request
- Trial: 1 free requests per buyer
- Billing order: trial requests -> subscribed plan requests -> pay as you go. 1 credit's USD value is shown on the marketplace page.

## HTTP routes

Only the public, buyer-facing parts of each route are listed; vendor-side credentials stay on the gateway.

### POST /crawls — Crawls
- Gateway: `POST https://api.apiclay.com/v1/api/commoncrawlwebarchiveindexapi/crawls`
- Upstream proxy: ApiClay proxy paid by the vendor (data cap 2 MB per call). Fetched through an ApiClay proxy paid by the vendor; a response larger than the route's data cap fails with 502 and is refunded in full.

### POST /lookup — URL lookup
- Gateway: `POST https://api.apiclay.com/v1/api/commoncrawlwebarchiveindexapi/lookup`
- Public query params: `url` (string, required), `matchType` (string), `limit` (number), `filter` (string), `from` (string), `to` (string)
- Public body fields: `crawl` (string, required)
- Body sample (JSON):

```json
{
  "crawl": "<crawl>"
}
```

- Upstream proxy: ApiClay proxy paid by the vendor (data cap 2 MB per call). Fetched through an ApiClay proxy paid by the vendor; a response larger than the route's data cap fails with 502 and is refunded in full.

## Call example

```bash
curl -X POST "https://api.apiclay.com/v1/api/commoncrawlwebarchiveindexapi/crawls" \
  -H "token: <YOUR_APICLAY_TOKEN>"
```

Index of every published API: https://apiclay.com/llms.txt
