AI web scraper builder: build extractors with your agent

What the agent does

1. 1

   Agree on the page and fields

   The agent asks only what it needs: an example URL of the page type and the fields you want.

2. 2

   Inspect the page

   It fetches the page and picks the cheapest fetch mode that works: plain HTML, embedded JSON, a JSON API, JavaScript rendering, XHR capture or browser actions.

3. 3

   Write the definition

   Request options plus field selectors, with stable selectors, typed values (prices as numbers, absolute URLs) and camelCase field names.

4. 4

   Validate

   validate_extractor_config runs the real extraction engine on the HTML it fetched, without fetching again and without credit; the agent fixes errors until every field has the right value.

5. 5

   Create and test the draft

   create_extractor saves a private draft; the agent then runs extract with the @draft pin on the real page (a normal, charged extract call that it announces first) and compares the result.

6. 6

   Release when you agree

   release_extractor with dry_run shows the planned version and schema change; after your OK it releases 1.0.0 and gives you the pins to call (@user/slug@1).

1. Install the skill

git clone https://github.com/extracthub-io/extracthub-extractor-builder.git ~/.claude/skills/extracthub-extractor-builder

# update later
git -C ~/.claude/skills/extracthub-extractor-builder pull

git clone https://github.com/extracthub-io/extracthub-extractor-builder.git
zip -r extracthub-extractor-builder.zip extracthub-extractor-builder -x "*.git*"

2. Connect the ExtractHub MCP server

Recommended: with the MCP server the agent validates, uploads, tests and releases by itself. Create an API key on the API keys page, then in Claude Code:

claude mcp add --transport http extracthub https://api.extracthub.io/mcp --header "Authorization: Bearer YOUR_EXTRACTHUB_API_KEY"

Claude Desktop, Cursor and other clients: see MCP setup. Use a live key to create and release extractors; a test key (exth_test_...) can only validate.

3. Ask for an extractor

• “Create an ExtractHub extractor for this shop's search result pages: title, price, rating and product link. https://www.example.com/search?q=laptop”
• “Scrape this job board with ExtractHub: title, company, location and salary for every job on https://jobs.example.com/listings”
• “My ExtractHub extractor @me/product-page returns empty prices on this page, fix it and release a patch.”
• “Turn this JSON API into an ExtractHub extractor that returns id, name and price for every item.”

At the end the agent reports the extractor id and pins, the fetch mode it chose and why, the fields with real sample values, the price tier (standard or JavaScript) and anything fragile it noticed.

Multi-step scrapes: the pipeline skill

For a list or category page plus every detail page it links to, install the companion ExtractHub pipeline builder skill (also MIT). The agent builds each extractor with this skill, then chains the released extractors into a pipeline, validates it, tells you what a run costs, runs it small once with your OK, checks the results and sets a schedule only when you agree. Running a pipeline spends credit and needs a live key.

git clone https://github.com/extracthub-io/extracthub-pipeline-builder.git ~/.claude/skills/extracthub-pipeline-builder

Get the pipeline skill on GitHub →

Costs, keys and visibility

• Validating, creating, updating and releasing use no credit (they count toward your per-minute request limit).
• A live test of the draft is a normal extract call, charged like any other; the agent tells you before it runs one.
• New extractors are private drafts. The agent asks before releasing a version or making an extractor unlisted or public.
• Custom headers are only sent on direct requests (useProxy: false), never through the scraping proxy, so the skill keeps proxied extractors header-free and uses direct requests for JSON APIs that need an auth header.

FAQ