How to scrape JavaScript rendered pages: when you really need a headless browser
How to scrape JavaScript rendered pages: check the raw HTML first, read embedded JSON, capture XHR responses, add waits and browser actions only when needed.
- Published
- Reading time
- 9 min read
- By
- ExtractHub team
- JavaScript rendering
- headless browser
- XHR
- browser actions
- web scraping
On this page
A page looks empty to your scraper, the browser shows it full of data, and the obvious fix is a headless browser. Sometimes that is right. Often it is not: the data was in the response all along, in a script tag or a JSON endpoint, and the browser only adds cost, latency and another moving part.
This post is a decision procedure to scrape JavaScript rendered pages without paying for a browser you do not need: check the raw HTML first, then embedded JSON, then the page’s own API, and only then render. When rendering is the answer, it shows how to wait for the content, capture background responses and run browser actions in an ExtractHub extractor.
Step 1: look at the raw HTML, not the browser
Your browser’s DevTools Elements panel shows the DOM after JavaScript ran, so it hides the question you are asking. Save the page the way a scraper fetches it, with a plain HTTP request:
curl -sL -A "Mozilla/5.0" "https://www.example.com/search?q=laptop" -o samples/raw.html
Then search the file for a value you can see on screen, such as a product title or a price (grep -c "Example laptop" samples/raw.html). Three outcomes:
- The value is in the markup. A plain fetch is enough. Write CSS selectors against this raw file.
- The value is in a
<script>block, as JSON-LD,__NEXT_DATA__or awindow.__STATE__ = {...}assignment. Still a plain fetch. Read the JSON instead of the markup (next step). - The file is a shell: an empty
<div id="root">, skeleton placeholders, a bundle of script tags and no data. Now you are looking at a page that may need rendering, but check steps 2 and 3 first.
One caution: a fetch from your own machine is an approximation. If curl gets an error page, a challenge or a “please enable JavaScript” notice while a browser loads fine, that says the site treats plain requests differently, not that it needs a browser. A fetch through the proxy (useProxy: true) often receives the full HTML without rendering. Test that before you switch to rendering.
Step 2: embedded JSON is usually the answer
Most sites built with a JavaScript framework render on the server and ship the data as JSON in the same response, so the browser can hydrate the page. That JSON is in the raw HTML. An extractor field with extract: "jsonContent" reads it with a JSON path, and the request stays on a plain-fetch tier.
The details (single values, lists from a JSON array, pages with several JSON-LD blocks) are in Scrape embedded JSON. For this decision it is enough to know that “the page is built with React” is not a reason to render it.
Step 3: call the page’s API directly
If the HTML is a true shell, the page fetches its data after it loads. Open DevTools, Network tab, filter by Fetch/XHR, and reload. Look for a response with the data you want, often a URL with /api/ or graphql in it.
If that URL answers a plain request (try it with curl), point an extractor at it with sourceType: "json" and read the fields with JSON paths. It needs no browser, and an API response tends to change less often than a page layout. Two things to check:
- Does it need auth headers? Custom headers are only sent on direct requests (
useProxy: false); the proxy never forwards them. An API that needs a header must be fetched directly, and that only works when it answers requests from a server. - Does it need browser cookies or a token the page computes? Then the direct call will not work, and the browser is the way to get the response (step 5).
Step 4: render the page, and wait for the content
When the data really only exists after JavaScript ran, set renderJavascript: true. ExtractHub loads the page in a headless browser and runs the extractor on the rendered DOM.
The most common failure with rendered pages is extracting too early: the page fired its load event, but the results are still loading. Tell the browser what to wait for. The simplest form is two request options:
waitForSelector: a CSS selector. The browser waits until that element exists, for at most 15 seconds.waitTime: an extra pause in milliseconds after that, capped at 15000.
Both run right after the page loads, before any browser actions. They never fail the request: if the element never appears, the extraction runs on the page as it is, and your required fields tell you that something went wrong. They only apply with renderJavascript; the validator warns requires_render if you set them on a plain fetch.
An illustrative example (the selectors fit an imaginary single-page app, not a real site):
{
"name": "Example jobs - search results",
"url": "https://jobs.example.com/search?q=data+engineer",
"method": "GET",
"useProxy": true,
"renderJavascript": true,
"waitForSelector": "#results [data-testid=\"job-card\"]",
"extractorConfig": {
"fields": [
{
"name": "jobs",
"selector": "#results [data-testid=\"job-card\"]",
"isArray": true,
"extract": "innerHTML",
"required": true,
"subExtractor": {
"fields": [
{ "name": "title", "selector": "h2 a", "extract": "textContent", "transforms": ["normalizeWhitespace"], "required": true },
{ "name": "url", "selector": "h2 a[href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] },
{ "name": "postedAt", "selector": "time[datetime]", "extract": "attributeValue", "attributeName": "datetime" }
]
}
}
]
}
}
Wait for the element you extract, not for something that appears earlier, such as the search box. And mark the list required: when the content does not load, the request fails with required_fields_missing instead of returning an empty list that looks like “no results”.
Step 5: capture the background responses instead of the DOM
When the page loads its data from an API you cannot call directly (step 3), let the browser make the call and read the response. With captureXhr: true (plus renderJavascript), the content the extractor sees is not HTML but a JSON array of the background responses the page made:
[
{
"url": "https://jobs.example.com/api/search?q=data+engineer",
"method": "GET",
"status_code": 200,
"response_body": "{\"results\":[{\"title\":\"Data engineer\"}]}"
}
]
Set sourceType: "json", keep only the responses you want with xhrUrlFilter (comma-separated URL substrings such as "/api/search,graphql"), and read the body through a sub-extractor. The string body is parsed as JSON for you:
{
"captureXhr": true,
"renderJavascript": true,
"xhrUrlFilter": "/api/search",
"extractorConfig": {
"sourceType": "json",
"fields": [
{
"name": "search",
"jsonPath": "[0].response_body",
"subExtractor": { "fields": [ { "name": "titles", "jsonPath": "results[*].title" } ] }
}
]
}
}
The order of captured responses can vary between runs, so make the filter specific enough that index [0] is always the response you mean. You get the API’s own field names and types, with no selectors to break when the layout changes.
Step 6: browser actions for clicks, typing and scrolling
Some content only appears after an interaction: a cookie banner covers the page, results load on “Show more”, a tab hides the specs. browserActions run in order in the browser before the HTML is taken:
| Action | Keys | Typical use |
|---|---|---|
wait_for_element | selector, timeout_s | Wait until the content you extract exists |
wait | wait_time_s | A fixed pause, as a last resort |
click | selector | Cookie banners, “load more”, tabs |
input | selector, value | Type into a field |
scroll | x, y, optional selector | Scroll by a number of pixels |
scroll_to_bottom | timeout_s, optional selector | Lazy loading and infinite scroll |
A selector here is an object, { "type": "css", "value": "..." }, and also accepts xpath and text (visible text). These selectors run in a real browser, so any CSS works, unlike extractor field selectors, which use a narrower syntax. Each action takes on_error: "error" (the default) fails the request, "skip" continues. Use skip for steps that do not always apply:
"browserActions": [
{ "type": "wait_for_element", "selector": { "type": "css", "value": "#results [data-testid=\"job-card\"]" }, "timeout_s": 10 },
{ "type": "click", "selector": { "type": "css", "value": "button#accept-cookies" }, "on_error": "skip" },
{ "type": "wait", "wait_time_s": 1, "on_error": "skip" },
{ "type": "scroll_to_bottom", "timeout_s": 5, "on_error": "skip" }
]
Keep the sequence short: every wait adds to the time the request takes, and long chains of waits and timeouts risk running out of time. scroll_to_bottom with a selector scrolls an inner panel instead of the window, with max_scrolls, pause_s and stop_when to bound it. Element scrolling cannot run on every site; when it cannot run for a request, the call fails before any page is fetched with 422 browser_action_unavailable, and it is not charged.
Before you script clicks to reach data that loads on demand, check that request in DevTools. If the site answers it with an error to automated browsers, more actions will not fix that.
What rendering changes: price, proxy and headers
Rendering is a request option with consequences beyond speed:
- Price tier. Every extract call is billed per URL at one of three tiers: direct (plain fetch, no proxy), proxy (plain fetch through the proxy) and JavaScript.
renderJavascript,captureXhror any browser action puts the extractor on the JavaScript tier. A longer scroll does not cost more, but each call does. Prices are on the pricing section. - Always through the proxy. JavaScript rendering runs through ExtractHub’s proxy, whatever
useProxysays. Saving an extractor with rendering storesuseProxy: true, and the validator warnsproxy_enabled_for_jsif you set it to false. - No custom headers, no cookies. Since the proxy never forwards custom headers, a rendered request sends none,
Cookieincluded. A page that only shows its data to a logged-in session cannot be scraped this way. - GET only. Do not combine a POST body with rendering: not every rendering route can send one.
The middle ground: allowJsEscalation
Some pages come back fine most of the time and occasionally as an empty body or a block page. Rendering every call to cover the bad ones is expensive. allowJsEscalation: true keeps the extractor on a plain fetch through the proxy and retries once with JavaScript rendering when the response is empty or looks blocked. Only the escalated calls are billed at the JavaScript tier, and their result metadata says "escalated": true. The balance check before each call covers the JavaScript price, since any call might escalate.
Escalation is decided from the fetched response, not from your fields: a server-rendered shell that returns a normal page without the data does not escalate. For those pages, use renderJavascript or one of steps 2 and 3. A direct request (useProxy: false) never escalates.
Validate on the rendered DOM, test on the live page
An extractor is validated on a sample of the page, and the sample must match the fetch mode:
- A plain-fetch extractor is validated on the raw HTML only. A DOM saved from a browser contains elements added by JavaScript: selectors on them pass the dry run and come back empty in production.
- A rendered extractor is validated on the rendered DOM, saved after the element you wait for appeared, with the same clicks and scrolls as your browser actions. A Playwright script with
page.waitForSelectorandpage.content()does it. Never take it from a logged-in browser profile.
Comparing the two files is also the proof that you need rendering: data present only in the rendered one. The extractor builder skill follows this procedure when Claude Code builds an extractor, as described in How to build a web scraper with Claude Code.
The dry run uses no credit. The final word is one live test of the draft through the extract API with the @draft pin, which runs with the exact request options of the extractor and is charged at its tier. When you release, the request options, including rendering and browser actions, become part of the immutable version: later edits to the draft change nothing for callers until you release again.
Checklist to scrape JavaScript rendered pages
- Saved the raw HTML with a plain request and searched it for a real value.
- Checked the script tags for JSON-LD,
__NEXT_DATA__or window state. - Looked in the Network tab for an API the page calls, and tried it directly.
- Tried a plain fetch through the proxy when your own request was blocked.
- If rendering: wait for the element you extract, mark the list
required, keep browser actions short and optional steps onskip. - Validated on the rendered DOM and ran one live
@drafttest.
The request options are listed in the API reference, and an agent can run the same extractors over MCP.
FAQ
How do I know if a page needs JavaScript rendering to scrape?
Save the raw HTML with a plain HTTP request (curl) and search it for a value you want, such as a product title or a price. If it is there, or in a script block of JSON, a plain fetch is enough. If the raw HTML is an empty shell and the value only appears in a browser, you need rendering.
Is scraping with a headless browser more expensive?
Yes. In ExtractHub, JavaScript rendering, XHR capture and browser actions all use the JavaScript price tier, which costs more than a plain fetch, and a rendered page takes longer to load. The current prices are on the pricing section of the home page.
What is XHR capture in web scraping?
The headless browser records the background responses the page loads, such as calls to a search API, and the extractor reads that JSON instead of the HTML. In ExtractHub you turn on captureXhr with renderJavascript, narrow the responses with xhrUrlFilter and set sourceType to json.
Can I send cookies or custom headers with a rendered page?
No. JavaScript rendering always goes through the proxy, and custom headers, Cookie included, are never sent through the proxy. Headers are only sent on direct requests without rendering.
Can I scrape a JavaScript page without a headless browser?
Often, yes. Many framework sites ship the data as JSON in the raw HTML (JSON-LD, __NEXT_DATA__ or window state), or load it from an API you can call directly. Check the raw HTML and the Network tab before you turn on rendering.
Related
- CSS selectors that survive a redesign: robust selectors for web scraping: Robust CSS selectors for scraping: test ids, itemprop and semantic tags over generated classes, attributes over display text, scoping and fallbacks.
- Web scraping sandbox: test a scraping integration without spending credit: Use a web scraping sandbox to test your integration: test API keys, sandbox_scenario failures, async jobs, paused batches and signed webhooks, at no cost.
- Batch web scraping: thousands of URLs with one API call: How a batch web scraping API works: submit up to 1000 URLs, poll progress, page through NDJSON results with a cursor, get signed webhooks and cancel.
- Agent skill: install the extractor builder skill for Claude Code
- MCP server: connect Claude Code, Cursor or any MCP client
- API reference: realtime extract, async jobs and the Batch API
- Web scraper catalog: public extractors you can call today