{ "openapi": "3.1.0", "info": { "title": "Extractor API", "version": "1.0.0", "description": "**Extractor API** - Extract structured data from any website\n\nFetch HTML from URLs and parse them into structured JSON using CSS selectors.\n\n### Features\n- **Extractor Studio** - Easily build and test extractors\n- **JavaScript Rendering** - Render dynamic pages with JavaScript\n- **Custom Headers & POST** - Full control over HTTP requests\n- **Structured Output** - Get clean JSON data from messy HTML\n\n### Use Cases\n- Extract product data from e-commerce sites\n- Scrape article content and metadata\n- Parse tables and lists into structured format\n- Monitor web pages for changes\n- Build custom web scrapers and data pipelines\n\n### Extractor versions\n`extractor_id` must be pinned to a released version:\n- `@username/slug@1` - latest version in major 1 (recommended; receives fixes and additive changes)\n- `@username/slug@1.2.3` - exactly that version\n- `@username/slug@draft` - the current Studio draft (extractor owner only)\n\nAn unpinned id (`@username/slug`) is rejected with `400` and the list of available majors.\nResponses carry `X-Extractor-Version`; a deprecated major also gets `Deprecation` and `Sunset` headers.\n\n### Errors\nErrors share one body: `{ \"success\": false, \"error\": \"\", \"errors\": [{ \"code\", \"message\", \"path\"? }], \"requestId\" }`.\nBranch on `errors[].code`: `validation_error` (400), `invalid_target` (400, a direct-fetch URL or redirect pointing to a private,\ninternal or ExtractHub host), `unauthorized` (401), `insufficient_balance` (402, the message has the\nrequired amount and your balance; nothing is charged), `rate_limited` / `too_many_queued` (429), `upstream_error` /\n`target_error` (502), `browser_action_unavailable` (422, scrolling inside an element cannot run for this site right now;\nfree), `queue_timeout` / `no_provider_available` (503), `upstream_timeout` (504), `internal_error` (500).\nEndpoint-specific answers (`pin_required` 400, 403 / 404 / 410, batch and job 4xx) carry `{ success: false, error, details: { code } }`.\n\n### Batch API\nExtract up to 1000 URLs in one request with `POST /api/extractor/batch`. The batch runs in the background:\npoll `GET /api/extractor/batch/{id}` for progress and read `GET /api/extractor/batch/{id}/results` (NDJSON).\nOnly succeeded URLs are charged, at the same price as Extract; credit for the next URLs is held while they run.\n\n### Async jobs (202)\nA realtime Extract request still waiting for capacity after 30 s becomes an async job instead of failing:\n`202 { job_id, status, status_url, result_url }` with `Location` and `Retry-After`. The request runs with\npriority over batches; poll `GET /api/extractor/jobs/{id}` (result inline once completed) or\n`GET /api/extractor/jobs/{id}/result`. The job is charged only if it succeeds; `POST /api/extractor/jobs/{id}/cancel`\ncancels it if it has not started yet. Pass `async_fallback=false`\nto get `503 queue_timeout` with `Retry-After` instead, and `webhook_url` to be notified.\n\n### Webhooks\nBatches (`webhook_url`, `webhook_events`) and jobs (`webhook_url`) send these events as a JSON POST:\n`batch.completed`, `batch.failed`, `batch.paused`, `job.completed`, `job.failed`.\n\n**Account default webhook**: set a default URL, its events and an enabled switch on the API keys page of the app.\nA batch or job submitted without `webhook_url` uses it (if enabled, with its event selection); a request-level\n`webhook_url` always wins. The URL is fixed when the batch / job is created: changing the default later does not\nretarget batches or jobs already submitted.\n\n**Test event**: \"Send test event\" on the API keys page sends `webhook.test` to the saved default URL, signed and\nlogged like any event (one attempt, no retries), with `data: { \"message\": \"...\" }`.\n\n```json\n{ \"id\": \"evt_...\", \"type\": \"job.completed\", \"created_at\": \"2026-10-30T12:00:00.000Z\",\n \"data\": { \"job_id\": \"job_...\", \"status\": \"completed\", \"counts\": { \"total\": 1, \"succeeded\": 1, \"failed\": 0, \"cancelled\": 0, \"pending\": 0 },\n \"cost\": 0.002, \"status_url\": \"...\", \"result_url\": \"...\", \"metadata\": null } }\n```\n\nThe payload never contains result data: fetch it from `results_url` / `result_url`. Headers:\n`X-Webhook-Id` (event id, the same for every attempt: use it to deduplicate), `X-Webhook-Event`,\n`X-Webhook-Attempt` and `X-Webhook-Signature: t=,v1=`.\n\n**Verify the signature** with your webhook signing secret (`whsec_...`, API keys page of the app):\n1. Split the header on `,`: `t` is the timestamp, each `v1` is a signature (two during a secret rotation).\n2. Compute `HMAC-SHA256(secret, \".\")` as lowercase hex.\n3. Accept if it equals one of the `v1` values (constant-time compare) and `t` is within 5 minutes of now.\n\nAnswer with any `2xx` within 10 s. Redirects are not followed and count as failures. Failed deliveries are retried\n6 times over about 24 hours (after 1 min, 10 min, 1 h, 4 h, 18 h). URLs must be https and resolve to public addresses.\n\n### Sandbox (test API keys)\nCreate a **test key** on the API keys page of the app (\"Test key\"). Test keys start with `exth_test_` (live keys\nwith `exth_`) and work on every endpoint above, but never fetch a page and never use credit.\n\n**Same as live**: endpoints, request validation, `extractor_id` pin resolution and its `400` / `403` / `404` / `410`\nerrors (any extractor a live key can use), response shapes, batch / job lifecycle, status / results / cancel endpoints,\nresult retention, webhooks (really sent, signed with your secret, account default webhook included).\n\n**Different from live**:\n- Results are simulated: the pinned version's last passing health-check output, else its released sample output,\n else a deterministic placeholder built from its output schema (`metadata.sandboxSource` says which).\n- Every sandbox response has `X-Sandbox: true` and `\"sandbox\": true` (extract `result`, batch / job status, batch\n results, webhook `data`). Costs are `0`; nothing appears in billing or usage.\n- Own rate limit: 30 requests per minute per account (all test keys together); sandbox batches take at most 50 targets.\n\n**Scenarios**: `sandbox_scenario` (GET query, POST body, batch body, or per batch target) picks the outcome:\n\n| Scenario | Extract (GET / POST) | Batch |\n|---|---|---|\n| `success` (default) | `200` with the simulated result | every URL succeeds |\n| `failed` | `502 upstream_error` | URLs fail (`upstream_error`) |\n| `timeout_to_job` | `202` async job that completes right after; `503 queue_timeout` with `async_fallback=false` | like success |\n| `slow` | `200` after about 4 s | URLs take about 4 s each |\n| `insufficient_credit` | `402` | the first half of the URLs runs, then `paused_insufficient_credit` (`batch.paused`); it resumes at the first sweep at least 1 min after the pause (sweeps run every 5 min) and completes |\n| `insufficient_credit_expired` | `402` | pauses like `insufficient_credit`, then fails with `failure_reason: insufficient_credit` (`batch.failed`) |\n\nPer batch target only `success`, `failed`, `slow` and `timeout_to_job` are allowed. An unknown scenario is\n`400 validation_error`; a live key sending `sandbox_scenario` gets `400 validation_error` (it is never ignored).\n\n### Authentication\nAll endpoints require your API key, in the `X-API-Key` header or as `Authorization: Bearer `. Sending both with different keys is `400 ambiguous_credentials`.\n\n### Rate Limits\nStandard rate limits apply based on your account tier.", "contact": { "name": "Extractor Studio", "url": "https://dash.extracthub.io/app/studio", "email": "support@extracthub.io" }, "license": { "name": "MIT", "url": "https://opensource.org/licenses/MIT" } }, "servers": [ { "url": "https://api.extracthub.io", "description": "Production" }, { "url": "https://data-stage.datascraper.workers.dev", "description": "Staging" } ], "security": [ { "ApiKeyAuth": [] }, { "BearerAuth": [] } ], "tags": [ { "name": "Web Scraper", "description": "Web scraping and HTML parsing endpoints" }, { "name": "Batch", "description": "Extract many URLs in the background: submit, poll, read results, cancel" }, { "name": "Jobs", "description": "Realtime Extract requests that became async jobs (202): status, result and cancel" } ], "components": { "securitySchemes": { "ApiKeyAuth": { "type": "apiKey", "in": "header", "name": "X-API-Key", "description": "Your ExtractHub API key" }, "BearerAuth": { "type": "http", "scheme": "bearer", "description": "Your ExtractHub API key as the Bearer token" } }, "schemas": {}, "parameters": {} }, "paths": { "/api/extractor/extract": { "get": { "tags": [ "Extractor" ], "summary": "Extract (GET)", "description": "Fetch and extract structured data from any URL using a pre-configured Extractor.\nUse this endpoint to scrape and extract target pages with GET request.\n\nWhen no scraping capacity becomes free within 30 s, the request is accepted as an async job: `202` with\n`job_id`, `status_url` and `result_url` (also in `Location`). Poll the job or pass `webhook_url`\nto be notified. Opt out with `async_fallback=false` to get `503 queue_timeout` with `Retry-After` instead.\n\n**Billing**: charged per URL at the version's tier when it succeeds: `direct` (no proxy, no JavaScript), `premium` (through the\nscraping proxy) or `premium_js` (JavaScript rendering, always through the proxy). A request that fails after the scraping\nprovider fetched the page (`target_error`, `upstream_timeout`, extraction / schema errors after the fetch) is\ncharged in full; its error body then has `charged: true` and `cost`. Failures before any provider work\n(validation, `402`, `429`, `queue_timeout`, `no_provider_available`, `upstream_error`) are not charged.", "operationId": "1-extractor-get", "parameters": [ { "schema": { "type": "string", "format": "uri", "description": "The URL to scrape and extract data from" }, "required": true, "description": "The URL to scrape and extract data from", "name": "target_url", "in": "query" }, { "schema": { "type": "string", "minLength": 1, "description": "Pinned extractor version: @username/slug@1 (latest in major 1), @username/slug@1.2.3 (exact) or @username/slug@draft (owner only). Unpinned ids are rejected.", "example": "@username/amazon-product@1" }, "required": true, "description": "Pinned extractor version: @username/slug@1 (latest in major 1), @username/slug@1.2.3 (exact) or @username/slug@draft (owner only). Unpinned ids are rejected.", "name": "extractor_id", "in": "query" }, { "schema": { "type": "string", "enum": [ "true", "false" ], "default": "false", "description": "Whether to include raw HTML in the response" }, "required": false, "description": "Whether to include raw HTML in the response", "name": "return_raw_html", "in": "query" }, { "schema": { "type": "string", "enum": [ "true", "false" ], "description": "Default true: a request still waiting for scraping capacity after 30 s becomes an async job (202 with job_id). false: answer 503 queue_timeout with Retry-After instead" }, "required": false, "description": "Default true: a request still waiting for scraping capacity after 30 s becomes an async job (202 with job_id). false: answer 503 queue_timeout with Retry-After instead", "name": "async_fallback", "in": "query" }, { "schema": { "type": "string", "format": "uri", "description": "HTTPS URL notified with job.completed / job.failed if the request becomes an async job (private and loopback addresses are rejected). When omitted, your account default webhook (API keys page) is used if enabled." }, "required": false, "description": "HTTPS URL notified with job.completed / job.failed if the request becomes an async job (private and loopback addresses are rejected). When omitted, your account default webhook (API keys page) is used if enabled.", "name": "webhook_url", "in": "query" }, { "schema": { "type": "string", "enum": [ "success", "failed", "timeout_to_job", "slow", "insufficient_credit", "insufficient_credit_expired" ], "description": "Test API keys only (sandbox): simulated outcome. success (default), failed (502 upstream_error), timeout_to_job (202 async job), slow (answers after a few seconds), insufficient_credit / insufficient_credit_expired (402 for Extract; batches pause). Rejected with 400 for live keys." }, "required": false, "description": "Test API keys only (sandbox): simulated outcome. success (default), failed (502 upstream_error), timeout_to_job (202 async job), slow (answers after a few seconds), insufficient_credit / insufficient_credit_expired (402 for Extract; batches pause). Rejected with 400 for live keys.", "name": "sandbox_scenario", "in": "query" } ], "responses": { "200": { "description": "Successfully extracted data from the web page", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ] } }, "required": [ "success", "result" ] } } } }, "202": { "description": "Accepted as an async job: no capacity became free within 30 s (see Jobs)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "job_id": { "type": "string", "description": "Job id" }, "status": { "type": "string", "enum": [ "queued" ], "description": "Job status, initially queued" }, "status_url": { "type": "string", "description": "GET this URL for the job status (includes the result once completed)" }, "result_url": { "type": "string", "description": "GET this URL for the result in the Extract response shape" }, "extractor_version": { "type": "string", "description": "Exact version the job runs with (e.g. 1.2.3, or draft)" }, "created_at": { "type": "string", "description": "ISO 8601 timestamp" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a job of a test API key (sandbox)" } }, "required": [ "job_id", "status", "status_url", "result_url", "extractor_version", "created_at" ] } }, "required": [ "success", "result" ] } } } }, "400": { "description": "Bad request - invalid URL, extractor_id or parameters. Validation failures are `validation_error` entries in `errors`; other 400s (`pin_required`, ...) carry `error` and `details`", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "target_url: Required", "errors": [ { "code": "validation_error", "message": "target_url: Required", "path": "target_url" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "402": { "description": "Payment required - your credit balance cannot cover this request (`insufficient_balance`; the message has the required amount and your balance). Nothing was charged; top up and retry", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry.", "errors": [ { "code": "insufficient_balance", "message": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "403": { "description": "Forbidden - insufficient permissions or credits", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "404": { "description": "Extractor Configuration not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "410": { "description": "Gone - the pinned extractor version was yanked", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit (`rate_limited`) or too many of your requests waiting for capacity (`too_many_queued`); retry after Retry-After", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "500": { "description": "Failed to extract data from the web page", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "502": { "description": "Bad gateway - the scrape provider failed (`upstream_error`) or the target page answered with an error status (`target_error`). `target_error` (the provider fetched the page) is charged and the body says `charged: true`; `upstream_error` is not charged. Retry later", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Unable to scrape at this time, please try again.", "errors": [ { "code": "upstream_error", "message": "Unable to scrape at this time, please try again." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "503": { "description": "`queue_timeout`: no capacity within 30 s and async_fallback=false (or the job could not be held), or `no_provider_available`; retry after Retry-After", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "No scraping capacity became available within 30s. Retry after 5s.", "errors": [ { "code": "queue_timeout", "message": "No scraping capacity became available within 30s. Retry after 5s." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "504": { "description": "Gateway timeout - the scrape provider did not answer in time (`upstream_timeout`). Charged (the provider usually did the work; the body says `charged: true`); retry later", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "The scraping service did not answer in time", "errors": [ { "code": "upstream_timeout", "message": "The scraping service did not answer in time" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } }, "post": { "tags": [ "Extractor" ], "summary": "Extract (POST)", "description": "Fetch HTML from any URL and parse it into structured JSON using pre-defined Extractor Configuration.\nUse this endpoint for targets that require a POST body or custom headers; for everything else use Extract (GET).\nIt takes the same pinned extractor_id as Extract (GET), e.g. @username/slug@1.\n\nWhen no scraping capacity becomes free within 30 s, the request is accepted as an async job: `202` with\n`job_id`, `status_url` and `result_url` (also in `Location`). Poll the job or pass `webhook_url`\nto be notified. Opt out with `\"async_fallback\": false` to get `503 queue_timeout` with `Retry-After` instead.\n\n**Billing**: charged per URL at the version's tier when it succeeds: `direct` (no proxy, no JavaScript), `premium` (through the\nscraping proxy) or `premium_js` (JavaScript rendering, always through the proxy). A request that fails after the scraping\nprovider fetched the page (`target_error`, `upstream_timeout`, extraction / schema errors after the fetch) is\ncharged in full; its error body then has `charged: true` and `cost`. Failures before any provider work\n(validation, `402`, `429`, `queue_timeout`, `no_provider_available`, `upstream_error`) are not charged.", "operationId": "2-extractor-post", "requestBody": { "content": { "application/json": { "schema": { "type": "object", "properties": { "target": { "type": "object", "properties": { "url": { "type": "string", "format": "uri", "description": "URL to scrape" }, "body": { "type": "string", "description": "Request body (for POST requests)" }, "headers": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Custom HTTP headers for the target (replace the version's saved headers and body). Sent only when the extractor version fetches directly (useProxy false); ignored when it goes through the scraping proxy. Custom headers are not sent when the request goes through the scraping proxy (useProxy true); they are only sent on direct requests (useProxy false). Scraping providers manage headers for ban avoidance. On direct requests (useProxy false) custom headers are sent as they are, Cookie included. Only add headers the target needs (e.g. API auth / Content-Type for JSON endpoints); overriding User-Agent or Accept* may get the request blocked." } }, "required": [ "url" ] }, "extractor_id": { "type": "string", "minLength": 1, "description": "Pinned extractor version: @username/slug@1 (latest in major 1), @username/slug@1.2.3 (exact) or @username/slug@draft (owner only). Unpinned ids are rejected.", "example": "@username/amazon-product@1" }, "options": { "type": "object", "properties": { "returnRawHtml": { "type": "boolean", "default": false, "description": "Include raw HTML in response" } }, "description": "Additional options" }, "async_fallback": { "type": "boolean", "description": "Default true: a request still waiting for scraping capacity after 30 s becomes an async job (202 with job_id). false: answer 503 queue_timeout with Retry-After instead" }, "webhook_url": { "type": "string", "format": "uri", "description": "HTTPS URL notified with job.completed / job.failed if the request becomes an async job (private and loopback addresses are rejected). When omitted, your account default webhook (API keys page) is used if enabled." }, "sandbox_scenario": { "type": "string", "enum": [ "success", "failed", "timeout_to_job", "slow", "insufficient_credit", "insufficient_credit_expired" ], "description": "Test API keys only (sandbox): simulated outcome. success (default), failed (502 upstream_error), timeout_to_job (202 async job), slow (answers after a few seconds), insufficient_credit / insufficient_credit_expired (402 for Extract; batches pause). Rejected with 400 for live keys." } }, "required": [ "target", "extractor_id" ] } } } }, "responses": { "200": { "description": "Successfully scraped and extract the web page", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ] } }, "required": [ "success", "result" ] } } } }, "202": { "description": "Accepted as an async job: no capacity became free within 30 s (see Jobs)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "job_id": { "type": "string", "description": "Job id" }, "status": { "type": "string", "enum": [ "queued" ], "description": "Job status, initially queued" }, "status_url": { "type": "string", "description": "GET this URL for the job status (includes the result once completed)" }, "result_url": { "type": "string", "description": "GET this URL for the result in the Extract response shape" }, "extractor_version": { "type": "string", "description": "Exact version the job runs with (e.g. 1.2.3, or draft)" }, "created_at": { "type": "string", "description": "ISO 8601 timestamp" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a job of a test API key (sandbox)" } }, "required": [ "job_id", "status", "status_url", "result_url", "extractor_version", "created_at" ] } }, "required": [ "success", "result" ] } } } }, "400": { "description": "Bad request - invalid URL, extractor_id or parameters. Validation failures are `validation_error` entries in `errors`; other 400s (`pin_required`, ...) carry `error` and `details`", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "target_url: Required", "errors": [ { "code": "validation_error", "message": "target_url: Required", "path": "target_url" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "402": { "description": "Payment required - your credit balance cannot cover this request (`insufficient_balance`; the message has the required amount and your balance). Nothing was charged; top up and retry", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry.", "errors": [ { "code": "insufficient_balance", "message": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "403": { "description": "Forbidden - insufficient permissions or credits", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "404": { "description": "Not found - resource does not exist", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "410": { "description": "Gone - the pinned extractor version was yanked", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit (`rate_limited`) or too many of your requests waiting for capacity (`too_many_queued`); retry after Retry-After", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "500": { "description": "Failed to scrape or parse the web page", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "502": { "description": "Bad gateway - the scrape provider failed (`upstream_error`) or the target page answered with an error status (`target_error`). `target_error` (the provider fetched the page) is charged and the body says `charged: true`; `upstream_error` is not charged. Retry later", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Unable to scrape at this time, please try again.", "errors": [ { "code": "upstream_error", "message": "Unable to scrape at this time, please try again." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "503": { "description": "`queue_timeout`: no capacity within 30 s and async_fallback=false (or the job could not be held), or `no_provider_available`; retry after Retry-After", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "No scraping capacity became available within 30s. Retry after 5s.", "errors": [ { "code": "queue_timeout", "message": "No scraping capacity became available within 30s. Retry after 5s." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "504": { "description": "Gateway timeout - the scrape provider did not answer in time (`upstream_timeout`). Charged (the provider usually did the work; the body says `charged: true`); retry later", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "The scraping service did not answer in time", "errors": [ { "code": "upstream_timeout", "message": "The scraping service did not answer in time" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/batch": { "post": { "tags": [ "Batch" ], "summary": "Create batch", "description": "Submit up to 1000 URLs for one pinned extractor version. The batch runs in the background; poll status_url\nfor progress and read the results from results_url.\n\n- **Billing**: succeeded URLs are charged at the same per-URL price as Extract, and so are failed URLs whose scrape\n reached the provider (target error status, block page, extraction error, provider timeout), once per URL. URLs that\n fail before any provider work (provider-side errors, no provider) and cancelled URLs cost nothing. Target-side\n errors are retried once. Credit for the next 100 URLs is held while they run and the unused part is released.\n- **402** when your balance cannot cover the first 100 URLs. When credit runs out later, the batch pauses\n (`paused_insufficient_credit`) and continues automatically after a top-up; after 72 hours it is stopped.\n- **Idempotency-Key** header (optional, max 255 characters): sending the same key again returns the\n existing batch (200) instead of creating a second one.\n- Per URL, `body` / `headers` replace the extractor version's saved request body and headers (headers are sent only when the version fetches directly, useProxy false; never through the scraping proxy).\n- **Webhooks**: `webhook_url` receives `batch.completed`, `batch.failed` and `batch.paused` (filter with\n `webhook_events`), signed with your webhook signing secret (see the API description). Without\n `webhook_url`, your account default webhook (API keys page) is used if enabled, with its event selection.\n- **Sandbox**: with a test API key the batch runs without fetching or charging anything (max 50 targets); pick\n outcomes with `sandbox_scenario` (batch) and `targets[].sandbox_scenario` (per URL). See \"Sandbox\" in the API description.", "operationId": "3-batch-create", "parameters": [ { "schema": { "type": [ "string", "null" ], "description": "Unique key per batch; a repeated key returns the existing batch" }, "required": false, "description": "Unique key per batch; a repeated key returns the existing batch", "name": "Idempotency-Key", "in": "header" } ], "requestBody": { "content": { "application/json": { "schema": { "type": "object", "properties": { "extractor_id": { "type": "string", "minLength": 1, "description": "Pinned extractor version, same rules as Extract: @username/slug@1, @username/slug@1.2.3 or @username/slug@draft (owner only). The version is resolved once at submission and used for every URL of the batch.", "example": "@username/amazon-product@1" }, "targets": { "type": "array", "items": { "type": "object", "properties": { "url": { "type": "string", "format": "uri", "description": "URL to scrape and extract" }, "body": { "type": "string", "maxLength": 65536, "description": "Request body for this URL. When body or headers are given, they replace the extractor version's saved body/headers" }, "headers": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Custom HTTP headers for this URL. Sent only when the extractor version fetches directly (useProxy false); ignored through the scraping proxy" }, "sandbox_scenario": { "type": "string", "enum": [ "success", "failed", "slow", "timeout_to_job" ], "description": "Test API keys only: outcome of this URL (overrides the batch sandbox_scenario). Rejected with 400 for live keys." } }, "required": [ "url" ] }, "minItems": 1, "maxItems": 10000, "description": "URLs to extract, 1 to 1000 per batch" }, "webhook_url": { "type": "string", "format": "uri", "description": "HTTPS URL notified about batch events (signed, retried for about 24 h). Private, loopback and link-local addresses are rejected. When omitted, your account default webhook (API keys page) is used if enabled." }, "webhook_events": { "type": "array", "items": { "type": "string", "enum": [ "batch.completed", "batch.failed", "batch.paused" ] }, "minItems": 1, "description": "Events sent to webhook_url (default: all). Requires webhook_url (the account default webhook uses its own event selection)" }, "metadata": { "type": "object", "additionalProperties": {}, "description": "Your own data, returned with the batch (max 4 KB as JSON)" }, "sandbox_scenario": { "type": "string", "enum": [ "success", "failed", "timeout_to_job", "slow", "insufficient_credit", "insufficient_credit_expired" ], "description": "Test API keys only (sandbox): success (default), failed (every URL fails), slow (URLs take a few seconds), insufficient_credit (the batch pauses after the first half of the URLs and resumes at the next sweep, within ~1-6 min), insufficient_credit_expired (pauses, then fails with insufficient_credit at the next sweep). timeout_to_job behaves like success. Rejected with 400 for live keys." } }, "required": [ "extractor_id", "targets" ] } } } }, "responses": { "200": { "description": "A batch with this Idempotency-Key already exists", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "batch_id": { "type": "string", "description": "Batch id" }, "status": { "type": "string", "description": "Batch status, initially queued" }, "status_url": { "type": "string", "description": "GET this URL for progress" }, "results_url": { "type": "string", "description": "GET this URL for the results (NDJSON)" }, "total_items": { "type": "number", "description": "Number of targets" }, "created_at": { "type": "string", "description": "ISO 8601 timestamp" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a batch of a test API key (sandbox)" } }, "required": [ "batch_id", "status", "status_url", "results_url", "total_items", "created_at" ] } }, "required": [ "success", "result" ] } } } }, "202": { "description": "Batch accepted", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "batch_id": { "type": "string", "description": "Batch id" }, "status": { "type": "string", "description": "Batch status, initially queued" }, "status_url": { "type": "string", "description": "GET this URL for progress" }, "results_url": { "type": "string", "description": "GET this URL for the results (NDJSON)" }, "total_items": { "type": "number", "description": "Number of targets" }, "created_at": { "type": "string", "description": "ISO 8601 timestamp" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a batch of a test API key (sandbox)" } }, "required": [ "batch_id", "status", "status_url", "results_url", "total_items", "created_at" ] } }, "required": [ "success", "result" ] } } } }, "400": { "description": "Bad request - invalid targets, webhook_url, metadata or unpinned extractor_id", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "402": { "description": "Payment required - your balance cannot cover the credit hold for the first chunk of URLs (`insufficient_balance`). Nothing was created or charged", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry.", "errors": [ { "code": "insufficient_balance", "message": "Insufficient balance: USD 0.0100 is required, your balance is USD 0.0040. Top up your credit and retry." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "403": { "description": "Forbidden - @draft is only available to the extractor owner", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "404": { "description": "Extractor not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "410": { "description": "Gone - the pinned extractor version was yanked", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "500": { "description": "Internal server error", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } } } } }, "/api/extractor/batch/{id}": { "get": { "tags": [ "Batch" ], "summary": "Get batch status", "description": "Progress of a batch: status, counts per item state, cost charged so far and timestamps.", "operationId": "4-batch-status", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Batch id" }, "required": true, "description": "Batch id", "name": "id", "in": "path" } ], "responses": { "200": { "description": "Batch status", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "batch_id": { "type": "string" }, "extractor_id": { "type": "string", "description": "Extractor id as submitted" }, "extractor_version": { "type": "string", "description": "Version the batch runs with (e.g. 1.2.3, or draft)" }, "status": { "type": "string", "enum": [ "queued", "running", "paused_insufficient_credit", "completed", "failed", "cancelled" ], "description": "queued: waiting to start; running; paused_insufficient_credit: waiting for credit (resumes automatically after a top-up); completed: every URL finished (some may have failed); failed: stopped early (see failure_reason); cancelled" }, "pause_reason": { "type": [ "string", "null" ], "description": "insufficient_credit while paused, else null" }, "failure_reason": { "type": [ "string", "null" ], "description": "Why the batch stopped early: insufficient_credit (paused too long), version_yanked, extractor_unavailable" }, "counts": { "type": "object", "properties": { "total": { "type": "number" }, "succeeded": { "type": "number" }, "failed": { "type": "number" }, "cancelled": { "type": "number" }, "pending": { "type": "number", "description": "Not finished yet (waiting or running)" } }, "required": [ "total", "succeeded", "failed", "cancelled", "pending" ] }, "cost": { "type": "number", "description": "USD charged so far (succeeded URLs only)" }, "metadata": { "type": [ "object", "null" ], "additionalProperties": {} }, "webhook_url": { "type": [ "string", "null" ] }, "created_at": { "type": "string" }, "updated_at": { "type": "string" }, "paused_at": { "type": [ "string", "null" ] }, "completed_at": { "type": [ "string", "null" ] }, "cancelled_at": { "type": [ "string", "null" ] }, "expires_at": { "type": [ "string", "null" ], "description": "Results are deleted after this time" }, "status_url": { "type": "string" }, "results_url": { "type": "string" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a batch of a test API key (sandbox): nothing is fetched or charged" } }, "required": [ "batch_id", "extractor_id", "extractor_version", "status", "pause_reason", "failure_reason", "counts", "cost", "metadata", "webhook_url", "created_at", "updated_at", "paused_at", "completed_at", "cancelled_at", "expires_at", "status_url", "results_url" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Batch not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/batch/{id}/results": { "get": { "tags": [ "Batch" ], "summary": "Get batch results", "description": "Finished URLs in index order: one JSON object per line (NDJSON, default) with `index`, `url`, `status`\n(succeeded / failed / cancelled), `result` or `error`, and `cost`.\n\nPaginate with `limit` and `cursor`: the `X-Next-Cursor` response header (NDJSON) or `next_cursor`\n(`format=json`) holds the cursor of the next page (treat it as opaque and pass it back unchanged). It is absent\nonly on the last page, once every URL after the cursor has finished and been returned.\n\nYou can read results while the batch is running. URLs finish in any order, but results are returned in index order\nwithout gaps: a page stops before the first URL that has not finished yet, so it may be shorter than `limit` or\nempty. Such a page still has a cursor (and `Retry-After`): wait and request the same cursor again. Following the\ncursor until it is absent returns every URL exactly once. Results are deleted 7 days after the batch ended (410).", "operationId": "5-batch-results", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Batch id" }, "required": true, "description": "Batch id", "name": "id", "in": "path" }, { "schema": { "type": "string", "pattern": "^(-1|\\d+)$", "description": "next_cursor of the previous page (X-Next-Cursor header for NDJSON), passed back unchanged; omit for the first page" }, "required": false, "description": "next_cursor of the previous page (X-Next-Cursor header for NDJSON), passed back unchanged; omit for the first page", "name": "cursor", "in": "query" }, { "schema": { "type": "integer", "minimum": 1, "maximum": 500, "default": 100, "description": "Items per page (max 500)" }, "required": false, "description": "Items per page (max 500)", "name": "limit", "in": "query" }, { "schema": { "type": "string", "enum": [ "ndjson", "json" ], "default": "ndjson", "description": "ndjson (default): one JSON object per line; json: { items, next_cursor }" }, "required": false, "description": "ndjson (default): one JSON object per line; json: { items, next_cursor }", "name": "format", "in": "query" } ], "responses": { "200": { "description": "Results page", "content": { "application/x-ndjson": { "schema": { "type": "object", "properties": { "index": { "type": "number", "description": "Position of the URL in targets" }, "url": { "type": "string" }, "status": { "type": "string", "enum": [ "succeeded", "failed", "cancelled" ] }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ], "description": "Extraction result (succeeded)" }, "error": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "description": "Why the URL failed or was cancelled" }, "cost": { "type": "number", "description": "USD charged for this URL" } }, "required": [ "index", "url", "status", "cost" ] } }, "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "items": { "type": "array", "items": { "type": "object", "properties": { "index": { "type": "number", "description": "Position of the URL in targets" }, "url": { "type": "string" }, "status": { "type": "string", "enum": [ "succeeded", "failed", "cancelled" ] }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ], "description": "Extraction result (succeeded)" }, "error": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "description": "Why the URL failed or was cancelled" }, "cost": { "type": "number", "description": "USD charged for this URL" } }, "required": [ "index", "url", "status", "cost" ] } }, "next_cursor": { "type": [ "string", "null" ] } }, "required": [ "items", "next_cursor" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Batch not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "410": { "description": "Gone - the results of this batch expired", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/batch/{id}/cancel": { "post": { "tags": [ "Batch" ], "summary": "Cancel batch", "description": "Stop a batch: URLs that have not started are cancelled (not charged) and their held credit is released.\nURLs already running finish and are charged if they succeed. Cancelling an ended batch changes nothing.", "operationId": "6-batch-cancel", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Batch id" }, "required": true, "description": "Batch id", "name": "id", "in": "path" } ], "responses": { "200": { "description": "Batch status after the cancel", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "batch_id": { "type": "string" }, "extractor_id": { "type": "string", "description": "Extractor id as submitted" }, "extractor_version": { "type": "string", "description": "Version the batch runs with (e.g. 1.2.3, or draft)" }, "status": { "type": "string", "enum": [ "queued", "running", "paused_insufficient_credit", "completed", "failed", "cancelled" ], "description": "queued: waiting to start; running; paused_insufficient_credit: waiting for credit (resumes automatically after a top-up); completed: every URL finished (some may have failed); failed: stopped early (see failure_reason); cancelled" }, "pause_reason": { "type": [ "string", "null" ], "description": "insufficient_credit while paused, else null" }, "failure_reason": { "type": [ "string", "null" ], "description": "Why the batch stopped early: insufficient_credit (paused too long), version_yanked, extractor_unavailable" }, "counts": { "type": "object", "properties": { "total": { "type": "number" }, "succeeded": { "type": "number" }, "failed": { "type": "number" }, "cancelled": { "type": "number" }, "pending": { "type": "number", "description": "Not finished yet (waiting or running)" } }, "required": [ "total", "succeeded", "failed", "cancelled", "pending" ] }, "cost": { "type": "number", "description": "USD charged so far (succeeded URLs only)" }, "metadata": { "type": [ "object", "null" ], "additionalProperties": {} }, "webhook_url": { "type": [ "string", "null" ] }, "created_at": { "type": "string" }, "updated_at": { "type": "string" }, "paused_at": { "type": [ "string", "null" ] }, "completed_at": { "type": [ "string", "null" ] }, "cancelled_at": { "type": [ "string", "null" ] }, "expires_at": { "type": [ "string", "null" ], "description": "Results are deleted after this time" }, "status_url": { "type": "string" }, "results_url": { "type": "string" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a batch of a test API key (sandbox): nothing is fetched or charged" } }, "required": [ "batch_id", "extractor_id", "extractor_version", "status", "pause_reason", "failure_reason", "counts", "cost", "metadata", "webhook_url", "created_at", "updated_at", "paused_at", "completed_at", "cancelled_at", "expires_at", "status_url", "results_url" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Batch not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/jobs/{id}": { "get": { "tags": [ "Jobs" ], "summary": "Get job status", "description": "Status of an async job. A realtime Extract request that waits too long for scraping capacity is answered with\n`202` and a `job_id` instead of `503`; the request runs as soon as capacity is free.\n\nOnce `completed`, `result` holds the extraction result (same as `result` of a `200` Extract response); a\n`failed` job has `error`. The job is charged at the Extract price when it completes, or when it fails after the provider fetched the page (same rule as Extract). Results are kept\nfor 7 days (`expires_at`); afterwards the status stays readable without `result`.", "operationId": "7-job-status", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Job id (job_...)" }, "required": true, "description": "Job id (job_...)", "name": "id", "in": "path" } ], "responses": { "200": { "description": "Job status", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "job_id": { "type": "string" }, "extractor_id": { "type": "string", "description": "Extractor id as requested" }, "extractor_version": { "type": "string", "description": "Exact version the job runs with" }, "status": { "type": "string", "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "description": "queued: waiting for capacity; running; completed: result available; failed: see error; cancelled: cancelled before it started (not charged)" }, "url": { "type": "string", "description": "Target URL" }, "cost": { "type": "number", "description": "USD charged (0 unless completed)" }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ], "description": "Extraction result, same as the result of a 200 Extract response (completed)" }, "error": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "description": "Why the job failed (failed), or code cancelled (cancelled)" }, "webhook_url": { "type": [ "string", "null" ] }, "created_at": { "type": "string" }, "updated_at": { "type": "string" }, "completed_at": { "type": [ "string", "null" ] }, "expires_at": { "type": [ "string", "null" ], "description": "The result is deleted after this time (status stays readable, result_url answers 410)" }, "status_url": { "type": "string" }, "result_url": { "type": "string" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a job of a test API key (sandbox)" } }, "required": [ "job_id", "extractor_id", "extractor_version", "status", "url", "cost", "webhook_url", "created_at", "updated_at", "completed_at", "expires_at", "status_url", "result_url" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Job not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/jobs/{id}/result": { "get": { "tags": [ "Jobs" ], "summary": "Get job result", "description": "The result of a job in the Extract response shape:\n- `200` completed: `{ success: true, result }`, exactly the body a `200` Extract response would have had\n- `202` still queued or running: the job status, with `Retry-After`\n- `422` failed or cancelled: `{ success: false, error, details: { code } }` (`code` is `cancelled` for a cancelled job)\n- `410` the result expired (7 days after completion)", "operationId": "8-job-result", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Job id (job_...)" }, "required": true, "description": "Job id (job_...)", "name": "id", "in": "path" } ], "responses": { "200": { "description": "Extraction result", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ] } }, "required": [ "success", "result" ] } } } }, "202": { "description": "The job has not finished yet", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "job_id": { "type": "string" }, "extractor_id": { "type": "string", "description": "Extractor id as requested" }, "extractor_version": { "type": "string", "description": "Exact version the job runs with" }, "status": { "type": "string", "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "description": "queued: waiting for capacity; running; completed: result available; failed: see error; cancelled: cancelled before it started (not charged)" }, "url": { "type": "string", "description": "Target URL" }, "cost": { "type": "number", "description": "USD charged (0 unless completed)" }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ], "description": "Extraction result, same as the result of a 200 Extract response (completed)" }, "error": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "description": "Why the job failed (failed), or code cancelled (cancelled)" }, "webhook_url": { "type": [ "string", "null" ] }, "created_at": { "type": "string" }, "updated_at": { "type": "string" }, "completed_at": { "type": [ "string", "null" ] }, "expires_at": { "type": [ "string", "null" ], "description": "The result is deleted after this time (status stays readable, result_url answers 410)" }, "status_url": { "type": "string" }, "result_url": { "type": "string" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a job of a test API key (sandbox)" } }, "required": [ "job_id", "extractor_id", "extractor_version", "status", "url", "cost", "webhook_url", "created_at", "updated_at", "completed_at", "expires_at", "status_url", "result_url" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Job not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "410": { "description": "Gone - the result of this job expired", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "422": { "description": "The job failed or was cancelled (details.code says why)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } }, "/api/extractor/jobs/{id}/cancel": { "post": { "tags": [ "Jobs" ], "summary": "Cancel job", "description": "Cancel a job that has not started: it becomes `cancelled` (not charged) and its held credit is released.\nA job that is already running finishes (status stays `running` until then) and is charged like any job (on success, or on a failure after the provider fetched the page).\nCancelling a finished or already cancelled job changes nothing and answers its current status. No webhook is sent\nfor a cancel.", "operationId": "9-job-cancel", "parameters": [ { "schema": { "type": "string", "minLength": 1, "description": "Job id (job_...)" }, "required": true, "description": "Job id (job_...)", "name": "id", "in": "path" } ], "responses": { "200": { "description": "Job status after the cancel", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ true ] }, "result": { "type": "object", "properties": { "job_id": { "type": "string" }, "extractor_id": { "type": "string", "description": "Extractor id as requested" }, "extractor_version": { "type": "string", "description": "Exact version the job runs with" }, "status": { "type": "string", "enum": [ "queued", "running", "completed", "failed", "cancelled" ], "description": "queued: waiting for capacity; running; completed: result available; failed: see error; cancelled: cancelled before it started (not charged)" }, "url": { "type": "string", "description": "Target URL" }, "cost": { "type": "number", "description": "USD charged (0 unless completed)" }, "result": { "type": "object", "properties": { "extracted": { "type": "object", "additionalProperties": {}, "description": "Extracted JSON data from HTML" }, "metadata": { "type": "object", "properties": { "useProxy": { "type": "boolean", "description": "Whether a proxy was used for scraping" }, "renderJs": { "type": "boolean", "description": "Whether JavaScript was rendered" }, "url": { "type": "string", "description": "Original URL" }, "scrapedAt": { "type": "string", "description": "ISO 8601 timestamp" }, "costUsd": { "type": "number", "description": "Cost of the extraction in USD" }, "escalated": { "type": "boolean", "description": "True when a blocked HTTP fetch was retried with JavaScript rendering" }, "warnings": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ] }, "description": "Non-blocking findings. direct_fetch_looks_blocked: a direct fetch (useProxy false) returned a page that looks like a block, captcha or consent page; retry with useProxy: true" }, "sandboxSource": { "type": "string", "enum": [ "health_check", "released_sample", "schema_fixture" ], "description": "Sandbox only: where the result came from (the version's last passing health check, its released sample, or a fixture built from its output schema)" } }, "required": [ "useProxy", "renderJs", "url", "scrapedAt" ], "description": "Extract metadata" }, "rawHtml": { "type": "string", "description": "Raw HTML content (if requested)" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) when the request used a test API key: simulated result, nothing fetched or charged" } }, "required": [ "extracted", "metadata" ], "description": "Extraction result, same as the result of a 200 Extract response (completed)" }, "error": { "type": "object", "properties": { "code": { "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "description": "Why the job failed (failed), or code cancelled (cancelled)" }, "webhook_url": { "type": [ "string", "null" ] }, "created_at": { "type": "string" }, "updated_at": { "type": "string" }, "completed_at": { "type": [ "string", "null" ] }, "expires_at": { "type": [ "string", "null" ], "description": "The result is deleted after this time (status stays readable, result_url answers 410)" }, "status_url": { "type": "string" }, "result_url": { "type": "string" }, "sandbox": { "type": "boolean", "enum": [ true ], "description": "Present (true) for a job of a test API key (sandbox)" } }, "required": [ "job_id", "extractor_id", "extractor_version", "status", "url", "cost", "webhook_url", "created_at", "updated_at", "completed_at", "expires_at", "status_url", "result_url" ] } }, "required": [ "success", "result" ] } } } }, "401": { "description": "Unauthorized - invalid, expired or missing API key (`unauthorized`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Invalid or expired API key", "errors": [ { "code": "unauthorized", "message": "Invalid or expired API key" } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } }, "404": { "description": "Job not found", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string" }, "details": { "description": "Additional error details" } }, "required": [ "success", "error" ] } } } }, "429": { "description": "Too many requests - per-account rate limit exceeded (`rate_limited`)", "content": { "application/json": { "schema": { "type": "object", "properties": { "success": { "type": "boolean", "enum": [ false ] }, "error": { "type": "string", "description": "Message of the first entry in errors" }, "errors": { "type": "array", "items": { "type": "object", "properties": { "code": { "type": "string", "description": "Stable machine-readable code, e.g. validation_error, unauthorized, insufficient_balance, rate_limited, too_many_queued, upstream_error, upstream_timeout" }, "message": { "type": "string" }, "path": { "type": "string", "description": "Request field the error is about (validation_error)" } }, "required": [ "code", "message" ] }, "minItems": 1 }, "requestId": { "type": "string", "description": "Quote this when contacting support" } }, "required": [ "success", "error", "errors", "requestId" ], "example": { "success": false, "error": "Rate limit exceeded. Please try again later.", "errors": [ { "code": "rate_limited", "message": "Rate limit exceeded. Please try again later." } ], "requestId": "6f1d0c1e-5a8c-4a52-9a53-3d0f2f6f7a10" } } } } } } } } }, "webhooks": {} }