How to build a web scraper with Claude Code and ExtractHub

Ask Claude Code to build a web scraper and it will happily write one: a script with fetch, a parser and a handful of CSS selectors. It works on the page you tested. Then the site ships a new layout, the script returns empty strings, and nothing tells you until the data downstream looks wrong.

This tutorial shows another way to build a web scraper with Claude Code: the agent still does the work, but the result is an extractor, a versioned web scraper hosted by ExtractHub that you call through an API. We go from a prompt to a released version and a curl call, step by step.

Why build your Claude Code web scraper as an extractor

An extractor is a JSON definition: how to fetch a page (request options) and which fields to pull out of it (selectors and transforms). Claude Code writes it with the open-source ExtractHub extractor builder skill. What you get compared with a script the agent writes for you:

• Versions with a contract. Releasing an extractor creates an immutable semver version with an output schema. Callers pin it, so a removed or retyped field can never break them silently: it needs a new major version.
• Nothing to host. The same extractor runs as a realtime API call, as an async job, in the Batch API, in pipelines and over MCP.
• Checks before money moves. The agent validates the definition against the real extraction engine on a saved copy of the page, without fetching again and without credit.
• Health checks. A released version whose required fields stop matching gets flagged, so you hear about a layout change before your users do.

What you need

1. Claude Code, in a terminal.
2. An ExtractHub account and a live API key from the API keys page. Creating and releasing extractors need a live key. A test key (exth_test_...) can validate definitions and call every endpoint in sandbox mode, but it never fetches a real page.
3. The extractor builder skill (details on the agent skill page). Clone it into your personal skills directory so every project can use it:

git clone https://github.com/extracthub-io/extracthub-extractor-builder ~/.claude/skills/extracthub-extractor-builder

4. The ExtractHub MCP server (recommended). It gives Claude Code the tools to validate, create, test and release extractors by itself. Replace the placeholder with your key:

claude mcp add --transport http extracthub https://api.extracthub.io/mcp --header "Authorization: Bearer YOUR_EXTRACTHUB_API_KEY"

Without the MCP server the skill still works: the agent checks the definition with its bundled checker and hands you a file to import in Extractor Studio. The rest of this guide assumes the MCP server is connected.

Step 1: describe the page and the fields

Start with one real URL of the page type and the fields you want. Claude Code loads the skill when you ask for an ExtractHub extractor:

Create an ExtractHub extractor for this shop’s search result pages: title, price, rating and product link. https://www.example.com/search?q=laptop

The skill asks only for what is missing: a second example URL helps catch selectors that only work on one page, and for a list page it confirms that you want one object per result, plus the next-page link if the results paginate. It does not ask you about request options; it decides those next.

Step 2: the agent inspects the page and picks a fetch mode

The agent saves the page to a file the way ExtractHub would fetch it, with a plain HTTP request and no browser. That file becomes the validation sample. Then it picks the cheapest fetch mode that has the data:

• Plain HTML, when the fields are in the raw page. If the site is open, a direct request without a proxy is the cheapest tier.
• Embedded JSON, when the data sits in a <script> block such as JSON-LD or __NEXT_DATA__. Still a plain fetch, and often the most stable source.
• A JSON API, when the page loads its data from an endpoint the extractor can call directly.
• JavaScript rendering, only when the raw HTML is an empty shell. It runs through the proxy and costs the most, so the agent uses it last, with background-request capture or browser actions when the data needs them.

The choice sets the price tier of every call (direct, proxy or JavaScript). The skill reports which mode it chose and why. See pricing for the current price of each tier.

Step 3: the agent writes the definition

For a search page the definition is one list field with a selector per result and sub-fields relative to each result. An illustrative example (the selectors fit an imaginary shop, not a real site):

{
  "name": "Example shop - search results",
  "description": "Title, price, rating and product link of every result on an example.com search page.",
  "slug": "example-shop-search",
  "url": "https://www.example.com/search?q=laptop",
  "method": "GET",
  "useProxy": true,
  "renderJavascript": false,
  "extractorConfig": {
    "fields": [
      {
        "name": "products",
        "selector": "main [data-testid=\"search-result\"]",
        "isArray": true,
        "extract": "innerHTML",
        "required": true,
        "subExtractor": {
          "fields": [
            { "name": "title", "selector": "h2", "extract": "textContent", "transforms": ["normalizeWhitespace"], "required": true },
            { "name": "price", "selector": "[itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
            { "name": "rating", "selector": "[data-rating]", "extract": "attributeValue", "attributeName": "data-rating", "transforms": ["toNumber"] },
            { "name": "url", "selector": "h2 a[href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] }
          ]
        }
      },
      { "name": "nextPageUrl", "selector": "a[rel=\"next\"][href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] }
    ]
  }
}

A few rules the skill follows, and that are worth knowing when you review its work:

• Stable hooks first: ids, data-testid, itemprop and semantic tags instead of generated class names.
• Machine-readable values: prices as numbers, links made absolute, attributes such as content or datetime instead of display text.
• required only where it proves the page was parsed. Here the list itself and each result’s title. An empty required field fails the request, so optional values such as a rating stay optional.
• Stable camelCase names. Renaming a field after a release is a breaking change.

Step 4: validate without credit

Next the agent runs validate_extractor_config with the saved page. It runs the real extraction engine on that HTML, without fetching anything and without credit, and returns errors, warnings and what each field extracted. The skill ships a script that sends the sample file for the agent, so a large page is never pasted into a tool call:

node scripts/validate.mjs example-shop-search.json --html samples/page-1.html --url "https://www.example.com/search?q=laptop"

The agent fixes and repeats until there are no errors, every field has the expected value and the list has as many items as the page shows. A sample_empty warning means a field matched nothing on this sample; the agent fixes it or confirms it on a second page where the value exists.

Step 5: a private draft and one announced live test

When the dry run is right, the agent calls create_extractor. The extractor is saved as a private draft with the id @you/example-shop-search. Only you can see and run it until you change its visibility.

Then it tests the draft on the live page with the extract tool and the pin @you/example-shop-search@draft. This is a real fetch, charged like any extract call, so the agent checks your balance with get_account and tells you before it runs the test. It compares the live result with the dry run. If they differ (a challenge page, a different layout, content that needs JavaScript), it adjusts the definition with update_extractor and tests again.

Step 6: dry-run the release, then release

Nothing is released without your OK. The agent first calls release_extractor with dry_run. ExtractHub runs the release check on the sample page and answers with the planned version and the output schema, without releasing anything. Through the skill’s script:

node scripts/validate.mjs --release @you/example-shop-search --html samples/page-1.html

When you agree, it releases. The first version is 1.0.0. Releasing uses no credit, and the version is immutable: later edits to the draft change nothing for callers until the next release.

Step 7: call your extractor from any code

Every call pins a version. @you/example-shop-search@1 follows the latest 1.x release; @you/example-shop-search@1.0.0 freezes one exact version. A realtime call with curl:

curl -X POST "https://api.extracthub.io/api/extractor/extract" \
  -H "X-API-Key: $EXTRACTHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "extractor_id": "@you/example-shop-search@1",
    "target": { "url": "https://www.example.com/search?q=headphones" }
  }'

The answer carries the extracted data under result.extracted. An illustrative example of its shape:

{
  "success": true,
  "result": {
    "extracted": {
      "products": [
        { "title": "Example headphones", "price": 49.9, "rating": 4.5, "url": "https://www.example.com/p/123" }
      ],
      "nextPageUrl": "https://www.example.com/search?q=headphones&page=2"
    },
    "metadata": { "url": "https://www.example.com/search?q=headphones" }
  }
}

The X-Extractor-Version response header says which version ran, and the metadata includes the cost of the call. An unpinned id such as @you/example-shop-search is rejected with a 400 that lists the available majors.

Keeping the scraper working

Pinning and semver. The version number follows the output schema. Selector fixes that keep the schema are patches, a new field is a minor version, and a removed, renamed or retyped field is a new major. Callers on @1 get fixes and new fields, never a breaking change. To retire a major, deprecate it with a sunset date at least 30 days out: responses then carry Deprecation and Sunset headers.

Health checks. Released versions are probed on the version’s sample page. Public extractors are checked on a schedule; private ones when you turn checks on. A version whose required fields come back empty, or that extracts nothing, is marked broken after two failing probes in a row. Optional fields never fail a check, which is why Step 3 keeps required short.

Fixing. When a check flags a version, ask Claude Code: “My ExtractHub extractor @you/example-shop-search returns empty prices on this page, fix it and release a patch.” It goes through the same validate, test and release loop.

Next steps: batches, jobs and MCP

• Reuse what others built: browse the public web scraper catalog and call a released extractor without writing one.
• Many URLs: send them to the Batch API (POST /api/extractor/batch) and read the results as NDJSON pages, with signed webhooks when the batch completes.
• Busy capacity: a realtime request that waits too long for capacity can become an async job (202 with job_id, status_url and result_url) unless you pass async_fallback=false.
• Agents as callers: any MCP client can search extractors, read their output schema and run them through the MCP server, with the same prices and limits as REST.
• Tests without credit: a test API key answers every endpoint from sample results, never fetches a page and never uses credit.
• List page, then detail pages: the companion pipeline builder skill chains released extractors into a pipeline. See the agent skill page.

The full endpoint list is in the API reference.