Batch web scraping: thousands of URLs with one API call
How a batch web scraping API works: submit up to 1000 URLs, poll progress, page through NDJSON results with a cursor, get signed webhooks and cancel.
- Published
- Reading time
- 10 min read
- By
- ExtractHub team
- batch scraping
- web scraping API
- webhooks
- NDJSON
- extractors
On this page
You have a list of 800 product URLs and an extractor that turns one product page into JSON. The obvious loop, one realtime request per URL, works until it doesn’t: you hit the per-minute request limit, a few requests time out, your script crashes at URL 512 and you no longer know which ones were done. A batch web scraping API moves that loop to the server. You send the whole list once, the platform runs it in the background with retries, and you read the results when they are ready.
This post walks through ExtractHub’s Batch API end to end: what a batch is, the request and response shapes, reading results as NDJSON pages, signed webhooks, cancelling, and how credit is held and charged while a batch runs.
When a batch beats a loop of realtime calls
Realtime extract (POST /api/extractor/extract) answers one URL while you wait. It is the right call for a user clicking a button. For a list of URLs, a batch is usually better:
- One request instead of N. The whole list is a single call, so it costs one request against your per-minute limit, not hundreds.
- Retries are handled for you. A target-side error (a 5xx, 408 or 429 from the site, or a page that came back without its required fields) is retried once. Transient errors on our side are retried a few more times. You don’t write backoff code.
- Crash-safe progress. The batch lives on the server. If your worker dies, you poll the status URL again and continue reading results where you stopped.
- No queue pressure on realtime traffic. Batch work runs at the lowest priority of the shared capacity queue, below realtime requests, so a big batch does not slow down your interactive calls.
The trade-off is latency: a batch finishes when its URLs finish, not in one request round trip. If you need one answer now, use realtime extract. If a realtime request waits too long for capacity it can also turn into a one-URL async job by itself (see the API reference).
The batch web scraping API in four endpoints
| Call | What it does |
|---|---|
POST /api/extractor/batch | Submit up to 1000 URLs for one pinned extractor version |
GET /api/extractor/batch/:id | Status, counts per item state, cost so far, timestamps |
GET /api/extractor/batch/:id/results | Finished URLs as NDJSON (or JSON), paginated by cursor |
POST /api/extractor/batch/:id/cancel | Cancel URLs that have not started; release their held credit |
All four take your API key as X-API-Key (or Authorization: Bearer). Every call, including status and results reads, counts as one request against your per-minute limit, and none of them is charged per request. You pay per URL, covered further down.
Step 1: submit the batch
A batch runs one extractor version over a list of targets. The extractor_id follows the same pinning rules as realtime extract: @you/slug@1 (latest 1.x release), @you/slug@1.2.3 (exact version) or @you/slug@draft (owner only). The pin is resolved once when you submit, so every URL in the batch runs the same version even if you release a new one halfway through. If you don’t have an extractor yet, build one with Claude Code or pick one from the public catalog.
An illustrative example with two URLs on example.com:
curl -X POST "https://api.extracthub.io/api/extractor/batch" \
-H "X-API-Key: $EXTRACTHUB_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: products-2026-11-14" \
-d '{
"extractor_id": "@you/example-product@1",
"targets": [
{ "url": "https://www.example.com/p/1001" },
{ "url": "https://www.example.com/p/1002" }
],
"webhook_url": "https://hooks.example.com/extracthub",
"webhook_events": ["batch.completed", "batch.failed"],
"metadata": { "run": "nightly-products" }
}'
The fields worth knowing:
targets: 1 to 1000 URLs. Each target can carry its ownbodyandheaders, which replace the version’s saved request body and headers for that URL. Custom headers are only sent when the version fetches directly (useProxy: false); through the scraping proxy they are not sent.Idempotency-Keyheader (optional, up to 255 characters): sending the same key again returns the existing batch with a 200 instead of creating a second one. Use it whenever your submit step can be retried, for example a key built from the job name and date.webhook_urlandwebhook_events: an HTTPS URL notified aboutbatch.completed,batch.failedandbatch.paused(all three by default). Private, loopback and link-local addresses are rejected. Withoutwebhook_url, your account default webhook from the API keys page is used if you enabled it.metadata: your own JSON (max 4 KB), returned with the status and in every webhook. Put the ids you need to match the batch back to your own records here.
The answer is a 202 with the batch id and the two URLs you will poll:
{
"success": true,
"result": {
"batch_id": "bat_...",
"status": "queued",
"status_url": "https://api.extracthub.io/api/extractor/batch/bat_...",
"results_url": "https://api.extracthub.io/api/extractor/batch/bat_.../results",
"total_items": 2,
"created_at": "2026-11-14T08:00:00.000Z"
}
}
A submission can also fail up front: 400 for bad targets, an invalid webhook URL, metadata over 4 KB or an unpinned extractor_id (the error lists the available majors), 403 for someone else’s @draft, 404 for an unknown extractor, 410 for a yanked version and 402 insufficient_balance when your balance cannot cover the first chunk of URLs (more on that below). In every one of those cases nothing is created and nothing is charged.
Step 2: watch progress
GET the status_url:
{
"success": true,
"result": {
"batch_id": "bat_...",
"extractor_id": "@you/example-product@1",
"extractor_version": "1.2.0",
"status": "running",
"pause_reason": null,
"failure_reason": null,
"counts": { "total": 2, "succeeded": 1, "failed": 0, "cancelled": 0, "pending": 1 },
"metadata": { "run": "nightly-products" },
"expires_at": null
}
}
(Trimmed: the response also has cost, the USD charged so far, plus the webhook URL, timestamps and both URLs.)
extractor_version tells you exactly which release the batch runs, which matters once you use versioned scrapers with a major pin. The status is one of:
queuedandrunning: work in progress.paused_insufficient_credit: waiting for credit; it resumes by itself after a top-up.completed: every URL finished. Some may have failed; checkcounts.failed.failed: the batch stopped early.failure_reasonsays why:insufficient_credit(paused too long),version_yanked(the pinned version was yanked while the batch ran; pending URLs are cancelled) orextractor_unavailable.cancelled: you cancelled it.
Poll at a calm interval, or skip polling entirely and wait for the webhook.
Step 3: page through NDJSON results
The results endpoint returns one JSON object per line (NDJSON), one line per finished URL, in the order of your targets:
{"index":0,"url":"https://www.example.com/p/1001","status":"succeeded","result":{"extracted":{"title":"Example product","price":49.9},"metadata":{"url":"https://www.example.com/p/1001"}},"cost":…}
{"index":1,"url":"https://www.example.com/p/1002","status":"failed","error":{"code":"http_404","message":"The target page returned HTTP 404"},"cost":…}
Again an illustrative example: the field names come from your extractor’s output schema, and cost (elided here) is the USD charged for that URL. index is the position in targets, so you can join results back to your input without matching URLs. Failed lines carry an error with a code and a message; cancelled lines too.
Two query parameters control paging: limit (1 to 500, default 100) and cursor. The cursor of the next page comes back in the X-Next-Cursor header (or next_cursor with format=json). Treat it as opaque and pass it back unchanged.
The part that makes results safe to read while the batch is still running: URLs finish in any order, but a page never skips one. It stops right before the first URL that has not finished yet. That page may be shorter than limit, or empty, and it still carries a cursor plus a Retry-After header. You wait and request the same cursor again. The cursor is absent only on the last page. Following it until it disappears returns every URL exactly once.
A reader in Node.js (18 or later, built-in fetch):
const API_URL = "https://api.extracthub.io";
const headers = { "X-API-Key": process.env.EXTRACTHUB_API_KEY };
async function readAllResults(batchId, onLine) {
let cursor;
for (;;) {
const url = new URL(`/api/extractor/batch/${batchId}/results`, API_URL);
url.searchParams.set("limit", "500");
if (cursor !== undefined) url.searchParams.set("cursor", cursor);
const res = await fetch(url, { headers });
if (!res.ok) throw new Error(`results: HTTP ${res.status}`);
const text = await res.text();
for (const line of text.split("\n")) if (line) onLine(JSON.parse(line));
const next = res.headers.get("X-Next-Cursor");
if (next === null) return; // last page: every URL was returned once
const retryAfter = res.headers.get("Retry-After");
if (retryAfter) await new Promise((r) => setTimeout(r, Number(retryAfter) * 1000));
cursor = next;
}
}
Store the last cursor you processed and a crashed reader can continue from it. Results are kept for 7 days after the batch ended; after that this endpoint answers 410 results_expired (the status endpoint keeps working).
Step 4: signed webhooks instead of polling
When a batch completes, fails or pauses, the webhook URL receives a small JSON event. It never contains result data: only ids, status, counts, the cost charged (a cost field, trimmed below), your metadata and the links to read the rest.
{
"id": "evt_...",
"type": "batch.completed",
"created_at": "2026-11-14T08:12:00.000Z",
"data": {
"batch_id": "bat_...",
"status": "completed",
"counts": { "total": 2, "succeeded": 1, "failed": 1, "cancelled": 0, "pending": 0 },
"status_url": "https://api.extracthub.io/api/extractor/batch/bat_...",
"results_url": "https://api.extracthub.io/api/extractor/batch/bat_.../results",
"metadata": { "run": "nightly-products" }
}
}
The request carries X-Webhook-Id, X-Webhook-Event, X-Webhook-Attempt and X-Webhook-Signature: t=<unix seconds>,v1=<hex>. The v1 value is an HMAC-SHA256 of <t>.<raw body> with your webhook signing secret, which you reveal and rotate on the API keys page. To accept an event, check that t is recent (a five-minute window is a sensible tolerance against replays), recompute the HMAC over the raw body, and compare in constant time. After a rotation the previous secret keeps signing for 24 hours, so the header can hold two v1 values: accept the event if either matches.
Answer with any 2xx. A redirect, a timeout or another status counts as a failed delivery, and failed deliveries are retried with growing delays for about a day. Because a retry can arrive after you already handled the event, deduplicate on X-Webhook-Id. The “Send test event” button on the API keys page sends a webhook.test event so you can check your receiver before the first real batch.
Cancelling a batch
POST /api/extractor/batch/:id/cancel cancels every URL that has not started. Cancelled URLs are not charged and their held credit is released. URLs already running finish normally and are charged by the usual rule. The answer is the batch status after the cancel; cancelling a batch that already ended changes nothing.
How credit is held and charged
A batch never charges you up front for the whole list. Credit moves in chunks of 100 URLs:
- On submit, credit for the first 100 URLs is held. If your balance cannot cover that hold, you get
402 insufficient_balancewith atop_up_url, and nothing is created. - While running, at most two chunks are held at a time. When a chunk finishes, its URLs are charged and the unused part of the hold is released.
- The hold per URL is the worst case: if the version allows JavaScript escalation, the hold covers the JavaScript price. A URL that did not escalate is charged at its own tier.
Which URLs are charged follows the same rule as realtime calls: a URL pays its per-URL price once, when it succeeds or when it fails after the page fetch reached the scraping service (a target 404, a block page, an empty required field). A URL that fails before any fetch work costs nothing, and so does a cancelled one. Retries never charge twice. The per-URL price is the extractor version’s tier (direct fetch, proxy or JavaScript); current prices are on the pricing section.
If credit runs out mid-batch, the batch finishes the URLs it already started, then pauses as paused_insufficient_credit and sends batch.paused. After a top-up it continues on its own. If it stays paused for 72 hours, it is stopped as failed with failure_reason: "insufficient_credit".
Test the integration before spending credit
With a test API key (exth_test_...) the same four endpoints work without fetching anything or moving credit, for up to 50 targets per batch. sandbox_scenario on the batch (success, failed, slow, insufficient_credit, insufficient_credit_expired) or on a single target lets you rehearse the unhappy paths, including the pause and its webhook. Sandbox events carry data.sandbox: true, so your receiver can tell them apart.
Batches from an AI agent
The MCP server exposes the same API as tools: create_batch, get_batch, get_batch_results and cancel_batch. They call the REST routes with your key, so limits, holds and charges are exactly the ones above. An agent that finds an extractor with search_extractors can run it over a list of URLs and read the results back without you writing the loop.
The full request and response schemas, error codes and the job endpoints are in the API reference.
FAQ
How many URLs can I send in one batch?
Up to 1000 URLs per batch with a live API key, and up to 50 with a test key. For more URLs, split them into several batches; an Idempotency-Key per batch makes a retried submission safe.
Are failed URLs in a batch charged?
A URL is charged once at the extractor's per-URL price when it succeeds, or when it fails after the page fetch reached the scraping service (for example the target answered 404 or a required field was empty). URLs that fail before any fetch work and cancelled URLs cost nothing.
What happens when my credit runs out during a batch?
The batch pauses with the status paused_insufficient_credit and sends a batch.paused webhook. It continues by itself after a top-up; if it stays paused for 72 hours it is stopped with failure_reason insufficient_credit.
Can I read batch results before the batch finishes?
Yes. The results endpoint returns finished URLs in index order and stops before the first unfinished one. The page then keeps a cursor and a Retry-After header, so you request the same cursor again later.
How long are batch results kept?
Seven days after the batch ended. After that the results endpoint answers 410 results_expired, while the status endpoint keeps working.
Related
- CSS selectors that survive a redesign: robust selectors for web scraping: Robust CSS selectors for scraping: test ids, itemprop and semantic tags over generated classes, attributes over display text, scoping and fallbacks.
- How to scrape JavaScript rendered pages: when you really need a headless browser: How to scrape JavaScript rendered pages: check the raw HTML first, read embedded JSON, capture XHR responses, add waits and browser actions only when needed.
- Web scraping sandbox: test a scraping integration without spending credit: Use a web scraping sandbox to test your integration: test API keys, sandbox_scenario failures, async jobs, paused batches and signed webhooks, at no cost.
- Agent skill: install the extractor builder skill for Claude Code
- MCP server: connect Claude Code, Cursor or any MCP client
- API reference: realtime extract, async jobs and the Batch API
- Web scraper catalog: public extractors you can call today