CSS selectors that survive a redesign: robust selectors for web scraping
Robust CSS selectors for scraping: test ids, itemprop and semantic tags over generated classes, attributes over display text, scoping and fallbacks.
- Published
- Reading time
- 9 min read
- By
- ExtractHub team
- CSS selectors
- web scraping
- reliability
- extractors
- tutorial
On this page
Most scrapers do not break because a site blocks them. They break because a selector stopped matching: a build tool renamed every class, a designer added a wrapper div, or a promo banner now sits above the price. The page still answers 200 and your scraper quietly returns null.
This guide is about robust CSS selectors for scraping: which hooks survive a redesign, why attributes beat display text, how to scope a selector so a new block on the page cannot steal its match, and how to keep a fallback while a site migrates. The examples are ExtractHub extractor fields, but the selector rules apply to any scraper.
Why selectors break in a redesign
Before choosing better selectors, it helps to know what actually changes when a site ships a new front end:
- Generated class names. CSS-in-JS libraries and CSS modules emit classes such as
css-1x9k2aborProductCard_title__a1b2c. The hash part changes whenever the styles or the build change, which can be every deploy. - Utility classes. A class list like
f6 b mr2 text-gray-700describes how something looks. Restyle the price in red and the selector is gone. - Positional chains.
div > div:nth-child(3) > spanencodes the exact tree. One extra wrapper, one removed column, and it points at something else, or at nothing. - First-match drift. A single-value field usually takes the first match in document order. When a “recently viewed” rail with the same
.priceclass appears above the product, your field now reads the wrong product. - Display text. Labels and formatted values change with language, A/B tests and marketing copy. “Price:” becomes “Our price”, “2 days ago” becomes a date.
Each of these has a counter-move. The rest of this post goes through them.
A ranking of robust CSS selectors for scraping
Not every hook is equal. Rank candidates by how likely they are to change for reasons that have nothing to do with you. From most to least stable:
| Rank | Hook | Example | Why it lasts |
|---|---|---|---|
| 1 | ids, test ids, data-* hooks | #product-title, [data-testid="price"] | Used by the site’s own tests and scripts; renaming them breaks their code too |
| 2 | Structured markup and semantic attributes | [itemprop="price"], a[rel="next"], input[name="sku"], [aria-label] | Exist for search engines, accessibility and forms |
| 3 | Semantic tags inside a scope | main h1, article time[datetime], table.specs td | Document structure changes less often than styling |
| 4 | Descriptive class names | .product-title, .price--current | Written by people, often stable, but owned by the design |
| 5 | Generated or utility classes | .css-1x9k2ab, .f6.b.mr2 | Change with any style or build change |
Test ids and data attributes
Many front ends put data-testid, data-test or data-qa attributes on the elements their end-to-end tests click and read. Those are some of the best hooks you can find: the team keeps them stable because their own tests depend on them. Other data-* attributes such as data-product-id or data-price are often read by the page’s scripts, which also keeps them around.
Search the raw HTML for data- near the value you want before you look at classes.
Structured markup and semantic attributes
Microdata (itemprop, itemscope), rel="next" on pagination links, name on form inputs and aria-label on icon buttons all exist for a purpose outside styling: search engines, accessibility tools and forms. A redesign that drops them hurts the site, so they tend to survive.
Semantic tags inside a scope
h1, article, time, table, dl / dd and nav describe the document. main h1 is a strong selector on most product and article pages. The scope (main, article, a test id on the content block) matters as much as the tag; see “Scope first” below.
Class names, last
Descriptive classes are fine when nothing better exists. Generated and utility classes are a last resort. If you must use one, combine it with a stable scope and expect maintenance.
Read attributes, not display text
Picking the element is half the job. The other half is reading the value from the place least likely to change. Machine-readable attributes are written for programs; display text is written for people and changes with them.
| Value | Fragile | Robust |
|---|---|---|
| Publish date | text “2 days ago” | time[datetime], attribute datetime |
| Price | text “Now only $49.90!” | [itemprop="price"][content], attribute content, or a data-price attribute |
| Next page | link text “Next ›“ | a[rel="next"][href], attribute href |
| In stock | text “In stock” vs “Available” | exists on a marker element such as .in-stock or button[disabled] |
| Image | src (often a lazy-load placeholder) | img[data-src], attribute data-src |
In an ExtractHub extractor that looks like this (illustrative example, the markup belongs to an imaginary page on example.com):
[
{ "name": "publishedAt", "selector": "article time[datetime]", "extract": "attributeValue", "attributeName": "datetime" },
{ "name": "price", "selector": "main [itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
{ "name": "nextPageUrl", "selector": "a[rel=\"next\"][href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] },
{ "name": "inStock", "selector": "main .in-stock", "extract": "exists" }
]
Two details worth copying:
- Put the attribute in the selector.
attributeValueon an element without that attribute returnsnull. Writingimg[data-src]instead ofimgskips those elements, so the first match is one that actually has the value. - Select the smallest element that holds only the value. A wrapper with a label (“Price: 49.90 EUR”) forces you to parse text. When the unit sits in its own child element,
excludeTagscan drop it fromtextContent.
Scope first, then select
Most “wrong value” bugs after a redesign are first-match drift. The fix is to scope every selector to the block you mean:
- Single values: prefix with the content container:
main h1,[data-testid="pdp"] [itemprop="price"],#content article time[datetime]. Headers, footers, sidebars and recommendation rails that reuse the same classes can then never come first. - Lists: the list selector must match each real item and nothing else. Scope it to the results container and exclude what is not a result:
#results > li:not(.sponsored)or[data-testid="search-result"]:not([data-ad]). Ads, skeleton placeholders and “you may also like” rails are the usual intruders. - Fields inside a list item: in an extractor list (
isArraywith asubExtractor), sub-field selectors run on each item’s inner HTML. Writeh3 a[href], not#results li h3 a[href]: the scope is already there, and a shorter selector has fewer parts that can change.
A quick check after any change: count the items on the page and compare with the length of the extracted array. If they differ, the list selector is either missing items or picking up intruders.
Before and after: one list rewritten
Here is a typical brittle extractor field next to a robust one. Illustrative example: the class names and attributes belong to an imaginary shop on example.com, not to a real site.
Brittle, copied from the browser’s “Copy selector”:
{
"name": "products",
"selector": "div.css-k008qs > div:nth-child(2) > div > div.css-1dbjc4n",
"isArray": true,
"extract": "innerHTML",
"subExtractor": {
"fields": [
{ "name": "title", "selector": "div.css-901oao > span" },
{ "name": "price", "selector": "span.css-16my406", "transforms": ["extractPrice"] }
]
}
}
Robust, built on hooks the page keeps for other reasons:
{
"name": "products",
"selector": "main [data-testid=\"search-result\"]:not([data-ad])",
"isArray": true,
"extract": "innerHTML",
"required": true,
"subExtractor": {
"fields": [
{ "name": "title", "selector": "h2", "transforms": ["normalizeWhitespace"], "required": true },
{ "name": "price", "selector": "[itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
{ "name": "url", "selector": "h2 a[href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] }
]
}
}
The second version has no generated class and no position, reads the price from a machine-readable attribute, excludes ads explicitly and marks the fields that prove the page was parsed as required. That last part matters later, when something still breaks.
Fallback selectors while a site migrates
Sites rarely switch layouts in one step. Old and new templates often run side by side for weeks, on different pages or for different visitors. A single selector then works on half the pages.
An extractor field can take a list of selectors with combineStrategy: "first". The engine tries them in order and the first one that yields a non-empty value wins. “Non-empty” is judged on the value, not on the element: a selector whose element exists but has empty text, or lacks the attribute, falls through to the next one.
{
"name": "title",
"selector": ["[data-testid=\"product-title\"]", "main h1"],
"combineStrategy": "first",
"transforms": ["normalizeWhitespace"]
}
Two rules apply. A selector array without combineStrategy returns null, so always set it. And a comma string such as "h1, h2" is not a fallback list: it is one selector that matches either, in document order. When the old template is gone, remove the dead selector so the definition says what the page actually is.
Selectors the extraction engine does not support
ExtractHub extracts on a streaming HTML parser, not in a browser. That lets plain-HTML pages skip the browser entirely, but it limits the selector syntax. Supported: tags, .class, #id, attribute selectors ([attr], ="v", ^=, $=, *=, ~=), compounds, descendant and child combinators, :nth-child(), :first-child, :nth-of-type(), :first-of-type, :not() with a simple selector, and comma lists.
Not supported: +, ~, :has(), :is(), :where(), :last-child, :last-of-type, :nth-last-child(), :only-child, :contains(), pseudo-elements and XPath. The validate_extractor_config tool of the MCP server reports a selector the engine cannot parse as an error, and the extractor builder skill’s bundled checker flags the same list before anything is uploaded.
The usual reason to reach for + or :has() is a value that only follows a label, as in <dt>Color</dt><dd>Red</dd>. Two options without them:
- Position among same-tag siblings:
#specs dl > dd:nth-of-type(2). Simple, but positions shift when a row is missing on some pages. - Read every row as a pair: a list of
{ label, value }objects over thedt/ddortrrows, and let your code map labels to fields. More robust when rows vary from page to page.
When the page has JSON, skip selectors
The most robust selector is often the one that points at a <script> block. Many sites ship the same data as JSON-LD, __NEXT_DATA__ or a window state object, and that JSON follows a data model rather than a design. A redesign can replace every class on the page and leave offers.price exactly where it was. The full method is in scrape embedded JSON. Cross-check the JSON against the visible page once; some sites leave stale values in it.
Catching the redesign that still gets through
No selector strategy is perfect. The goal is to break less often and to know at once when it happens.
- Mark the fields that prove the page was parsed as
required. An empty required top-level field fails the request withrequired_fields_missing(HTTP 422) instead of returning a record full ofnull. Mark the main listrequiredtoo: a required sub-field whose selector breaks drops every item, and an optional list would then come back as a quiet[]. - Use health checks. Public extractors are checked on a schedule, private ones when you turn checks on. A released version is probed on its sample page and judged on its required fields. How the probes decide between broken and degraded is in scraper monitoring.
- Ship the fix as a patch. A selector change that keeps the output schema is a patch release, so callers pinned to
@you/slug@1get it without touching their code. Why that works is in versioned web scrapers.
A checklist for review, whether you write the selectors or an agent does:
- No generated class names, unless scoped and accepted as maintenance.
- No positional chains longer than one step.
- Every single-value selector scoped to the content block.
- Values from attributes (
datetime,content,href,data-*) where the page has them. - List selectors that exclude ads and placeholders, with the item count checked.
- One or two
requiredfields per page type, and the main listrequired. - Validated on a second page of the same type before release.
Want to see how others solved a site? The public extractor catalog shows each extractor’s output fields, and you can call a released version instead of writing your own.
FAQ
What is the most stable CSS selector for web scraping?
Hooks that exist for reasons other than styling: an id, a data-testid or other data-* attribute, an itemprop, or attributes such as rel, name and aria-label. Semantic tags inside a scope (main h1, article time[datetime]) come next; generated class names come last.
Why does my scraper break when the site changes its CSS?
Usually because a selector depends on a generated class name, on a long positional chain, or on the first match in the document. A new build renames the classes, a new wrapper shifts the positions, or a new block above the content becomes the first match.
Should I scrape display text or attributes?
Attributes, where the page has them. A datetime, content or href attribute is written for programs, while display text such as "2 days ago" or "Next" changes with language, A/B tests and copy edits.
How do I add a fallback selector for when a site changes its layout?
In an ExtractHub extractor, give the field's selector as an array with combineStrategy "first": the first selector that yields a non-empty value wins, so an element that exists but is empty falls through to the next one.
Does :has() work in web scraping selectors?
Not in an ExtractHub extractor. Extraction runs on a streaming HTML parser, not in a browser, so +, ~, :has(), :is(), :where(), :last-child and similar selectors are not supported. Use :nth-of-type on same-tag siblings, or read label and value pairs as a list.
Related
- How to scrape JavaScript rendered pages: when you really need a headless browser: How to scrape JavaScript rendered pages: check the raw HTML first, read embedded JSON, capture XHR responses, add waits and browser actions only when needed.
- Web scraping sandbox: test a scraping integration without spending credit: Use a web scraping sandbox to test your integration: test API keys, sandbox_scenario failures, async jobs, paused batches and signed webhooks, at no cost.
- Batch web scraping: thousands of URLs with one API call: How a batch web scraping API works: submit up to 1000 URLs, poll progress, page through NDJSON results with a cursor, get signed webhooks and cancel.
- Agent skill: install the extractor builder skill for Claude Code
- MCP server: connect Claude Code, Cursor or any MCP client
- API reference: realtime extract, async jobs and the Batch API
- Web scraper catalog: public extractors you can call today