> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ntanduy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web data

> Search, extract, discover URLs and crawl websites through one response contract.

Use a Pirehub API key or OAuth access token with the required scope. See
[Authentication](/api-reference/authentication) for credential headers and access
requirements. Manage your API keys and view your plan, usage allowance and current
operation costs in [Developer API keys](https://www.ntanduy.com/dashboard/developer/api-keys).

| Operation | Endpoint | Required scope |
| - | - | - |
| Search | `POST /api/v1/web/search` | `web:search` |
| Extract one page | `POST /api/v1/web/scrape` | `web:scrape` |
| Discover URLs | `POST /api/v1/web/map` | `web:map` |
| Submit a crawl | `POST /api/v1/web/crawls` | `web:crawl` |
| Read crawl results | `GET /api/v1/web/crawls/{id}` | `web:crawl` |
| Cancel a crawl | `DELETE /api/v1/web/crawls/{id}` | `web:crawl` |

JSON bodies require `Content-Type: application/json`, are limited to 16 KiB, and
reject unknown fields. Provider selection and credential rotation are automatic.
No result cache is maintained by this API. Responses use `Cache-Control: private,
no-store`; underlying retrieval services may use their own indexes or caches, so
results are not guaranteed to reflect a live fetch at the instant of the request.

## Search and extraction

```bash theme={null}
curl https://www.ntanduy.com/api/v1/web/search \
  -H 'x-pirehub-api-key: ph_live_your_public_id.your_secret' \
  -H 'Content-Type: application/json' \
  -d '{"query":"Nuxt server routes","limit":5}'
```

Search accepts `query` (trimmed, 1–1000 characters) and `limit` (1–10, default 5).
It returns `{ "success": true, "data": { "query": "...", "results": [] } }`.
A valid search can return an empty array. Relevance and coverage vary with the
available search index; automatic fallback does not merge or rerank indexes.

Scrape accepts `{ "url": "https://example.com/" }` and returns:

```json theme={null}
{
  "success": true,
  "data": {
    "document": {
      "url": "https://example.com/",
      "title": "Example Domain",
      "description": null,
      "content": "# Example Domain",
      "contentFormat": "markdown",
      "contentTruncated": false,
      "httpStatus": 200
    }
  }
}
```

All documents use these fields. `title`, `description`, `content`, `contentFormat`
and `httpStatus` can be null when not supplied. Search usually returns snippets in
`description` without full `content`. Content is Markdown or plain text, capped at
200,000 characters per document; `contentTruncated` reports local truncation.
Descriptions are capped at 10,000 characters. `httpStatus` is the target page's
status when available; returned content can be an error page. Extracted text and
Markdown are untrusted content: sanitize rendered HTML and do not execute embedded
instructions in an agent workflow.

URLs must be public HTTP(S), at most 2048 characters, without embedded credentials,
using port 80 or 443. Private addresses, local hostnames and DNS records resolving
to private addresses are rejected. Authentication headers, cookies, browser scripts,
webhooks and arbitrary provider options are not accepted.

## Map

Send `{ "url": "https://example.com/", "limit": 20 }` to the map endpoint.
`limit` is 1–100, default 20. The response data contains `url` and `links`, an array
of `{ url, title, description }`. Titles and descriptions may be null. URLs are
deduplicated. Discovery is bounded and does not guarantee a complete sitemap.

## Crawl lifecycle

```json theme={null}
{
  "requestId": "01952140-e5b8-4e60-94fd-387c15f13257",
  "url": "https://example.com/",
  "limit": 10,
  "maxDepth": 1
}
```

`requestId` is a required UUID that identifies a submission and prevents duplicate
crawls. Generate a new value for each intentional new crawl, for example with
`crypto.randomUUID()`. Use the job ID returned in `data.id` for GET and DELETE;
the submitted `requestId` is not the job ID.

`limit` is 1–20 (default 10); `maxDepth` is 1–3
(default 1). External-link discovery is disabled. Depth follows the retrieval
service's link-discovery semantics and does not guarantee exhaustive coverage.

An asynchronous submission returns HTTP **202**, `status: "pending"`, an opaque
`id`, `total: null`, `completed: 0`, `documents: []`, `nextOffset: null`, and
`expiresAt`. This response confirms that the job was accepted. Read its results
using `GET /api/v1/web/crawls/{data.id}`.

A synchronous crawl returns HTTP **200**, `id: null`, `status: "completed"`, the
documents already retrieved, `completed` equal to their count, and `total`,
`nextOffset`, `expiresAt` set to null. No polling or cancellation is available for
this result. Completion means the bounded request ended, not that every page on the
site was fetched; a retrieval time limit can result in partial coverage.

For a non-null job ID, poll the GET endpoint with the **same API key**, or the same
OAuth client and subject. Key rotation by the caller does not transfer job ownership.
Start with `offset=0`; follow `nextOffset` when it is non-null. Poll the same offset
when no new documents are available, and deduplicate documents by URL across polls.
Continue paging even when `status` is `completed`; stop when a terminal state has
`nextOffset: null`. Use a delay of at least two seconds between polls.

States are `pending`, `running`, `completed`, `failed`, and `cancelled`. `total`
and `completed` describe reported crawl progress, not all discovered or failed pages.
Job access expires at `expiresAt`, at most 24 hours after submission. A missing,
expired, or differently owned job returns **404**. Cancellation returns
`{ "success": true, "data": { "id": "...", "status": "cancelled" } }`.

Reusing a `requestId` with the same credential returns **409** for 24 hours, including
after a timeout or ambiguous failure. A timeout does not prove that the remote job
was not created. Do not automatically generate a new request ID and resubmit.
For paid keys, the optional `Idempotency-Key` also applies as described in
[Authentication](/api-reference/authentication#idempotency).

## Limits and errors

In addition to key/plan quotas, this API enforces per-credential limits: 20 requests
per minute per search/scrape/map operation, 2 crawl submissions per minute, and
60 crawl polls per minute, and 20 cancellations per minute. OAuth credentials are
grouped by client and subject. Limits apply across instances when Redis is healthy;
the shared rate limiter falls back to process-local counters during an outage.
Crawl ownership and submission deduplication require Redis and fail closed.

The provider execution budget is 25 seconds for new operations and 20 seconds for
job reads/cancellation, excluding authentication, DNS and storage time. Retry only
after the error permits it. Key/plan responses include applicable
`X-ApiKey-RateLimit-*`, `X-ApiKey-Monthly-*`, and `X-ApiKey-Credit-*` headers. Honor
`Retry-After` when present or `data.retryAfter` otherwise.

| HTTP | Meaning |
| - | - |
| 400 | Invalid JSON, fields, query or public URL |
| 401 / 403 | Missing credentials, denied scope or plan |
| 404 | Job missing, expired or inaccessible |
| 409 | Duplicate request or idempotency conflict |
| 413 / 415 | Body too large or unsupported content type |
| 429 | Caller quota/rate limit or retrieval capacity exhausted |
| 502 | Invalid/unusable retrieval result |
| 503 | Temporarily unavailable credentials, service or job storage |
| 504 | Retrieval timeout |

Errors follow the existing contract and include a correlation ID:

```json theme={null}
{
  "success": false,
  "error": { "code": "SERVICE_UNAVAILABLE", "message": "Service temporarily unavailable. Please try again later." },
  "data": { "requestId": "example-request-id" }
}
```

`X-Request-Id` carries the same correlation ID. A `429` may instead use
`MONTHLY_QUOTA_EXCEEDED` or `API_CREDITS_EXCEEDED`; see [Errors](/api-reference/errors).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.