Scrape embedded JSON: read JSON-LD and __NEXT_DATA__ instead of HTML

Before you write a single CSS selector for a product page, open its source and search for <script. A lot of sites already ship the data you want as JSON: a schema.org JSON-LD block for search engines, the __NEXT_DATA__ script that hydrates a Next.js page, or a window.__INITIAL_STATE__ = {...} assignment from a single-page app. When you scrape embedded JSON instead of the rendered markup, you read typed values under keys the site’s own code depends on.

This post shows how to find embedded JSON in a page, how ExtractHub reads it with jsonContent, how to turn a JSON array into a clean list with jsonPath and [*], and where the approach stops working.

Why scrape embedded JSON instead of HTML

A CSS selector depends on how a page looks. Embedded JSON depends on what the page means. That difference shows up in four ways:

• Stability. Generated class names such as .sc-1x2y3z change with every build. The keys of a JSON-LD Product (name, offers.price, sku) are a public vocabulary, and the props in __NEXT_DATA__ are what the site’s own components read.
• Typed values. A price in JSON is often already a number, a date is often ISO 8601, a URL is often absolute. In the markup the same values are display text (“$1,299.00”, “3 days ago”) that you have to clean up.
• Completeness. The JSON often holds fields the template never shows: ids, SKUs, stock flags, every image size, the full description.
• Cost. The JSON is in the server’s HTML response, so a plain HTTP request is enough. No headless browser, no JavaScript rendering.

The trade-off: the JSON is an internal detail of the site too. A site can rename a prop in its state object just as it can rename a class. In practice, JSON-LD changes least (it exists for search engines), framework state changes when the site changes its data model, and class names change most often.

Step 1: find the JSON in the raw HTML

Look at the HTML the server sends, not at the DOM in your browser’s inspector: the inspector shows the page after JavaScript ran, which may contain things a plain fetch never gets. Save the raw response and search it:

curl -s "https://www.example.com/product/123" -o page.html
grep -o '<script[^>]*>' page.html | sort | uniq -c

The script tags worth checking:

Source	Selector	Typical content
schema.org JSON-LD	script[type="application/ld+json"]	Product, Article, JobPosting, ItemList, BreadcrumbList
Next.js (Pages Router)	script#__NEXT_DATA__	the page props under props.pageProps
Nuxt 3	script#__NUXT_DATA__	the page payload
Other inline JSON	script[type="application/json"][id="..."]	app config, initial data
Window state	script#__window__NAME (see below)	window.__INITIAL_STATE__ = {...} and similar

Then search for a value you can see on the page, such as the product title or the price. If it appears inside one of these scripts, you have found your source.

Step 2: read single values with jsonContent

In an ExtractHub extractor, an HTML field with extract: "jsonContent" parses the matched script’s text as JSON. With jsonPaths it returns the value at that path. An illustrative example for a product page with a JSON-LD Product block (the paths fit schema.org’s vocabulary, not a particular site):

{
  "sourceType": "html",
  "fields": [
    { "name": "name", "selector": "script[type=\"application/ld+json\"]", "extract": "jsonContent", "jsonPaths": ["name"], "required": true },
    { "name": "sku", "selector": "script[type=\"application/ld+json\"]", "extract": "jsonContent", "jsonPaths": ["sku"] },
    { "name": "price", "selector": "script[type=\"application/ld+json\"]", "extract": "jsonContent", "jsonPaths": ["offers.price"], "transforms": ["toNumber"] },
    { "name": "currency", "selector": "script[type=\"application/ld+json\"]", "extract": "jsonContent", "jsonPaths": ["offers.priceCurrency"] }
  ]
}

What the engine does with these paths:

• One path returns the value with its JSON type: number, string, object, array, boolean or null.
• Several paths return an object keyed by the full path text, for example {"offers.price": 1299, "offers.priceCurrency": "USD"}.
• No jsonPaths returns the whole parsed JSON, handy while exploring.
• Path syntax is dot notation with [n] indexes and quoted keys for odd names: props.pageProps["product-detail"].name, or ["@graph"][0].name for JSON-LD that wraps its nodes in @graph. A leading $. is accepted.
• Unparsable JSON or a missing path gives null, never an exception.

Keep toNumber on prices even when the JSON looks numeric: some sites write "price": "1299.00" as a string.

The NEXT_DATA version

A Next.js page that uses the Pages Router ships its props in <script id="__NEXT_DATA__" type="application/json">. The same product, as an illustrative example where the site’s props happen to be called product:

{ "name": "name", "selector": "script#__NEXT_DATA__", "extract": "jsonContent", "jsonPaths": ["props.pageProps.product.title"], "required": true }

The path below props.pageProps is the site’s own data model, so find it in your saved sample rather than guessing it.

Window state without a browser

Many apps assign their state in a plain script: window.__INITIAL_STATE__ = {...}, window["__APOLLO_STATE__"] = {...}, window.__DATA__ = JSON.parse('...') or var data = {...}. That is JavaScript, not a JSON script tag, so ExtractHub exposes each assignment as a virtual script you can select: script#__window__INITIAL_STATE, script#__window__APOLLO_STATE, script#__window__data (the name without leading and trailing underscores). Nothing is executed: only literals that are valid JSON are recovered, so a JavaScript object literal with unquoted keys or functions is skipped.

{ "name": "productId", "selector": "script#__window__INITIAL_STATE", "extract": "jsonContent", "jsonPaths": ["product.id"] }

Step 3: turn a JSON array into a list

A search or category page usually keeps its results in an array: an ItemList in JSON-LD, a product grid in __NEXT_DATA__, a results array in the window state. For that, give the field one jsonPath (singular) instead of jsonPaths. The rules of ExtractHub’s JSON engine then apply to the script’s JSON:

• [*] in the path walks every element of an array;
• subFields, each with its own jsonPath relative to the element, pick the properties you want;
• required on a sub-field drops elements that miss it;
• includePosition: true adds position (1..n) as the first key of each item, numbered after incomplete items were dropped.

An illustrative example for a category page with a JSON-LD ItemList:

{
  "name": "items",
  "selector": "script[type=\"application/ld+json\"]",
  "extract": "jsonContent",
  "jsonPath": "itemListElement[*]",
  "includePosition": true,
  "subFields": [
    { "name": "name", "jsonPath": "name", "required": true },
    { "name": "url", "jsonPath": "url", "required": true }
  ]
}

The output is an array of objects:

{
  "items": [
    { "position": 1, "name": "Example laptop 14", "url": "https://www.example.com/p/123" },
    { "position": 2, "name": "Example laptop 16", "url": "https://www.example.com/p/456" }
  ]
}

The same field shape works on __NEXT_DATA__ ("jsonPath": "props.pageProps.results[*]" in an illustrative site) and on window state. For nested lists inside each element, use subExtractor.fields instead of subFields: each element is then read by full JSON fields with their own [*] paths.

Two constraints of the list form: no isArray (the [*] makes the list) and no combineStrategy, and you cannot mix jsonPath with jsonPaths on one field.

Pages with several JSON-LD blocks

A real page often has three or four ld+json scripts: an Organization, a WebSite with a search action, a BreadcrumbList, then the block you want. The two forms treat that differently:

• A jsonPaths field reads the first matching script. If the Product block is not first, make the selector more specific or check the order in your sample.
• A jsonPath field tries the matched scripts in document order and uses the first one whose JSON parses and gives a non-empty value at the path. A block without itemListElement is skipped. A BreadcrumbList also has itemListElement, so if it comes before your ItemList, select the list block more precisely.

When a value can sit at two different paths (an A/B test, two page variants), give the field several jsonPaths and the coalesce transform: the first path that has a value wins. When it can sit in two different scripts, use a selector array with combineStrategy: "first": if the first script exists but its paths give nothing, the next script is tried.

{ "name": "title", "selector": "script#__NEXT_DATA__", "extract": "jsonContent",
  "jsonPaths": ["props.pageProps.product.title", "props.pageProps.item.name"], "transforms": ["coalesce"] }

Embedded JSON and the price of each call

An ExtractHub extractor’s request options decide its price tier. Rendering JavaScript (or capturing background requests, or browser actions) puts every call on the JavaScript tier. A plain fetch through the proxy is the middle tier, and a direct fetch without the proxy (useProxy: false) is the cheapest. The current prices are in the pricing section.

Embedded JSON keeps a page that “needs JavaScript” on a plain-fetch tier: the browser would only have run the script that turns the JSON into markup, and you read the JSON directly. A direct request also skips the realtime capacity queue, and a failed direct request is not charged. Use it for sites that answer a plain request; when a site blocks direct requests, the proxy tier still reads the same JSON.

Check the raw HTML first, though. If the server sends an empty shell and the data arrives later from an API call, there is no embedded JSON to read: call that JSON API directly with sourceType: "json", or capture the background response with JavaScript rendering.

Validate on a saved sample, then release

Test the fields on the page you saved, not on the live site. With the ExtractHub extractor builder skill the bundled script runs the real extraction engine on the file, without a fetch and without credit:

node scripts/validate.mjs example-category.json --html page.html --url "https://www.example.com/category/laptops"

It returns each field’s value and warns about fields that matched nothing on the sample. With the MCP server connected, an agent such as Claude Code runs the same check through validate_extractor_config, then creates, tests and releases the extractor; the Claude Code tutorial walks through that loop.

Two points matter for embedded JSON when you release:

• Output types come from the sample. A config does not fix the type of a jsonContent value, so the release takes it from the sample result (string, number, nested object or array). A value that is null on the sample stays unknown. Release with a sample where every field has a value.
• Keep required short. Mark the fields that prove the page parsed (a name, a URL) as required. An empty required field fails the request, and health checks judge only required fields, so an optional field that is missing on some products never flags the version.

Then call it like any other extractor, with a pinned version:

curl -X POST "https://api.extracthub.io/api/extractor/extract" \
  -H "X-API-Key: $EXTRACTHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "extractor_id": "@you/example-category@1",
    "target": { "url": "https://www.example.com/category/desktops" }
  }'

When embedded JSON is the wrong source

• The JSON is a summary. JSON-LD sometimes carries only a subset (one offer of many, the first page of reviews). Compare it with what the page shows before you rely on it.
• The values disagree with the page. A stale price in JSON-LD and a current price in the markup happen. If the visible value is the one you need, use a CSS selector for that field, ideally on an attribute such as itemprop="price" with content.
• The state is a JavaScript object literal. Unquoted keys or functions are not JSON; the window-state reader skips them. Fall back to the markup or to the site’s JSON API.
• The data is not in the first response. Then nothing in the HTML can help; see the JSON API and background capture options above.

Mixing sources in one extractor is fine: JSON-LD for the name, SKU and price, a CSS selector for a badge that only exists in the markup.

Next steps

• Browse the public extractor catalog to see released extractors and their output schemas.
• Read the endpoint details (realtime, jobs, batch) in the API reference.
• Install the extractor builder skill and let your agent check a page for embedded JSON before it writes any selector.