JSON bodies require
Content-Type: application/json, are limited to 16 KiB, and
reject unknown fields. Provider selection and credential rotation are automatic.
No result cache is maintained by this API. Responses use Cache-Control: private, no-store; underlying retrieval services may use their own indexes or caches, so
results are not guaranteed to reflect a live fetch at the instant of the request.
Search and extraction
query (trimmed, 1–1000 characters) and limit (1–10, default 5).
It returns { "success": true, "data": { "query": "...", "results": [] } }.
A valid search can return an empty array. Relevance and coverage vary with the
available search index; automatic fallback does not merge or rerank indexes.
Scrape accepts { "url": "https://example.com/" } and returns:
title, description, content, contentFormat
and httpStatus can be null when not supplied. Search usually returns snippets in
description without full content. Content is Markdown or plain text, capped at
200,000 characters per document; contentTruncated reports local truncation.
Descriptions are capped at 10,000 characters. httpStatus is the target page’s
status when available; returned content can be an error page. Extracted text and
Markdown are untrusted content: sanitize rendered HTML and do not execute embedded
instructions in an agent workflow.
URLs must be public HTTP(S), at most 2048 characters, without embedded credentials,
using port 80 or 443. Private addresses, local hostnames and DNS records resolving
to private addresses are rejected. Authentication headers, cookies, browser scripts,
webhooks and arbitrary provider options are not accepted.
Map
Send{ "url": "https://example.com/", "limit": 20 } to the map endpoint.
limit is 1–100, default 20. The response data contains url and links, an array
of { url, title, description }. Titles and descriptions may be null. URLs are
deduplicated. Discovery is bounded and does not guarantee a complete sitemap.
Crawl lifecycle
requestId is a required UUID that identifies a submission and prevents duplicate
crawls. Generate a new value for each intentional new crawl, for example with
crypto.randomUUID(). Use the job ID returned in data.id for GET and DELETE;
the submitted requestId is not the job ID.
limit is 1–20 (default 10); maxDepth is 1–3
(default 1). External-link discovery is disabled. Depth follows the retrieval
service’s link-discovery semantics and does not guarantee exhaustive coverage.
An asynchronous submission returns HTTP 202, status: "pending", an opaque
id, total: null, completed: 0, documents: [], nextOffset: null, and
expiresAt. This response confirms that the job was accepted. Read its results
using GET /api/v1/web/crawls/{data.id}.
A synchronous crawl returns HTTP 200, id: null, status: "completed", the
documents already retrieved, completed equal to their count, and total,
nextOffset, expiresAt set to null. No polling or cancellation is available for
this result. Completion means the bounded request ended, not that every page on the
site was fetched; a retrieval time limit can result in partial coverage.
For a non-null job ID, poll the GET endpoint with the same API key, or the same
OAuth client and subject. Key rotation by the caller does not transfer job ownership.
Start with offset=0; follow nextOffset when it is non-null. Poll the same offset
when no new documents are available, and deduplicate documents by URL across polls.
Continue paging even when status is completed; stop when a terminal state has
nextOffset: null. Use a delay of at least two seconds between polls.
States are pending, running, completed, failed, and cancelled. total
and completed describe reported crawl progress, not all discovered or failed pages.
Job access expires at expiresAt, at most 24 hours after submission. A missing,
expired, or differently owned job returns 404. Cancellation returns
{ "success": true, "data": { "id": "...", "status": "cancelled" } }.
Reusing a requestId with the same credential returns 409 for 24 hours, including
after a timeout or ambiguous failure. A timeout does not prove that the remote job
was not created. Do not automatically generate a new request ID and resubmit.
For paid keys, the optional Idempotency-Key also applies as described in
Authentication.
Limits and errors
In addition to key/plan quotas, this API enforces per-credential limits: 20 requests per minute per search/scrape/map operation, 2 crawl submissions per minute, and 60 crawl polls per minute, and 20 cancellations per minute. OAuth credentials are grouped by client and subject. Limits apply across instances when Redis is healthy; the shared rate limiter falls back to process-local counters during an outage. Crawl ownership and submission deduplication require Redis and fail closed. The provider execution budget is 25 seconds for new operations and 20 seconds for job reads/cancellation, excluding authentication, DNS and storage time. Retry only after the error permits it. Key/plan responses include applicableX-ApiKey-RateLimit-*, X-ApiKey-Monthly-*, and X-ApiKey-Credit-* headers. Honor
Retry-After when present or data.retryAfter otherwise.
Errors follow the existing contract and include a correlation ID:
X-Request-Id carries the same correlation ID. A 429 may instead use
MONTHLY_QUOTA_EXCEEDED or API_CREDITS_EXCEEDED; see Errors.