CSS selectors that survive a redesign: robust selectors for web scraping

Robust CSS selectors for scraping: test ids, itemprop and semantic tags over generated classes, attributes over display text, scoping and fallbacks.

Published
Reading time
9 min read
By
ExtractHub team
  • CSS selectors
  • web scraping
  • reliability
  • extractors
  • tutorial
On this page

Most scrapers do not break because a site blocks them. They break because a selector stopped matching: a build tool renamed every class, a designer added a wrapper div, or a promo banner now sits above the price. The page still answers 200 and your scraper quietly returns null.

This guide is about robust CSS selectors for scraping: which hooks survive a redesign, why attributes beat display text, how to scope a selector so a new block on the page cannot steal its match, and how to keep a fallback while a site migrates. The examples are ExtractHub extractor fields, but the selector rules apply to any scraper.

Why selectors break in a redesign

Before choosing better selectors, it helps to know what actually changes when a site ships a new front end:

  • Generated class names. CSS-in-JS libraries and CSS modules emit classes such as css-1x9k2ab or ProductCard_title__a1b2c. The hash part changes whenever the styles or the build change, which can be every deploy.
  • Utility classes. A class list like f6 b mr2 text-gray-700 describes how something looks. Restyle the price in red and the selector is gone.
  • Positional chains. div > div:nth-child(3) > span encodes the exact tree. One extra wrapper, one removed column, and it points at something else, or at nothing.
  • First-match drift. A single-value field usually takes the first match in document order. When a “recently viewed” rail with the same .price class appears above the product, your field now reads the wrong product.
  • Display text. Labels and formatted values change with language, A/B tests and marketing copy. “Price:” becomes “Our price”, “2 days ago” becomes a date.

Each of these has a counter-move. The rest of this post goes through them.

A ranking of robust CSS selectors for scraping

Not every hook is equal. Rank candidates by how likely they are to change for reasons that have nothing to do with you. From most to least stable:

RankHookExampleWhy it lasts
1ids, test ids, data-* hooks#product-title, [data-testid="price"]Used by the site’s own tests and scripts; renaming them breaks their code too
2Structured markup and semantic attributes[itemprop="price"], a[rel="next"], input[name="sku"], [aria-label]Exist for search engines, accessibility and forms
3Semantic tags inside a scopemain h1, article time[datetime], table.specs tdDocument structure changes less often than styling
4Descriptive class names.product-title, .price--currentWritten by people, often stable, but owned by the design
5Generated or utility classes.css-1x9k2ab, .f6.b.mr2Change with any style or build change

Test ids and data attributes

Many front ends put data-testid, data-test or data-qa attributes on the elements their end-to-end tests click and read. Those are some of the best hooks you can find: the team keeps them stable because their own tests depend on them. Other data-* attributes such as data-product-id or data-price are often read by the page’s scripts, which also keeps them around.

Search the raw HTML for data- near the value you want before you look at classes.

Structured markup and semantic attributes

Microdata (itemprop, itemscope), rel="next" on pagination links, name on form inputs and aria-label on icon buttons all exist for a purpose outside styling: search engines, accessibility tools and forms. A redesign that drops them hurts the site, so they tend to survive.

Semantic tags inside a scope

h1, article, time, table, dl / dd and nav describe the document. main h1 is a strong selector on most product and article pages. The scope (main, article, a test id on the content block) matters as much as the tag; see “Scope first” below.

Class names, last

Descriptive classes are fine when nothing better exists. Generated and utility classes are a last resort. If you must use one, combine it with a stable scope and expect maintenance.

Read attributes, not display text

Picking the element is half the job. The other half is reading the value from the place least likely to change. Machine-readable attributes are written for programs; display text is written for people and changes with them.

ValueFragileRobust
Publish datetext “2 days ago”time[datetime], attribute datetime
Pricetext “Now only $49.90!”[itemprop="price"][content], attribute content, or a data-price attribute
Next pagelink text “Next ›“a[rel="next"][href], attribute href
In stocktext “In stock” vs “Available”exists on a marker element such as .in-stock or button[disabled]
Imagesrc (often a lazy-load placeholder)img[data-src], attribute data-src

In an ExtractHub extractor that looks like this (illustrative example, the markup belongs to an imaginary page on example.com):

[
  { "name": "publishedAt", "selector": "article time[datetime]", "extract": "attributeValue", "attributeName": "datetime" },
  { "name": "price", "selector": "main [itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
  { "name": "nextPageUrl", "selector": "a[rel=\"next\"][href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] },
  { "name": "inStock", "selector": "main .in-stock", "extract": "exists" }
]

Two details worth copying:

  • Put the attribute in the selector. attributeValue on an element without that attribute returns null. Writing img[data-src] instead of img skips those elements, so the first match is one that actually has the value.
  • Select the smallest element that holds only the value. A wrapper with a label (“Price: 49.90 EUR”) forces you to parse text. When the unit sits in its own child element, excludeTags can drop it from textContent.

Scope first, then select

Most “wrong value” bugs after a redesign are first-match drift. The fix is to scope every selector to the block you mean:

  • Single values: prefix with the content container: main h1, [data-testid="pdp"] [itemprop="price"], #content article time[datetime]. Headers, footers, sidebars and recommendation rails that reuse the same classes can then never come first.
  • Lists: the list selector must match each real item and nothing else. Scope it to the results container and exclude what is not a result: #results > li:not(.sponsored) or [data-testid="search-result"]:not([data-ad]). Ads, skeleton placeholders and “you may also like” rails are the usual intruders.
  • Fields inside a list item: in an extractor list (isArray with a subExtractor), sub-field selectors run on each item’s inner HTML. Write h3 a[href], not #results li h3 a[href]: the scope is already there, and a shorter selector has fewer parts that can change.

A quick check after any change: count the items on the page and compare with the length of the extracted array. If they differ, the list selector is either missing items or picking up intruders.

Before and after: one list rewritten

Here is a typical brittle extractor field next to a robust one. Illustrative example: the class names and attributes belong to an imaginary shop on example.com, not to a real site.

Brittle, copied from the browser’s “Copy selector”:

{
  "name": "products",
  "selector": "div.css-k008qs > div:nth-child(2) > div > div.css-1dbjc4n",
  "isArray": true,
  "extract": "innerHTML",
  "subExtractor": {
    "fields": [
      { "name": "title", "selector": "div.css-901oao > span" },
      { "name": "price", "selector": "span.css-16my406", "transforms": ["extractPrice"] }
    ]
  }
}

Robust, built on hooks the page keeps for other reasons:

{
  "name": "products",
  "selector": "main [data-testid=\"search-result\"]:not([data-ad])",
  "isArray": true,
  "extract": "innerHTML",
  "required": true,
  "subExtractor": {
    "fields": [
      { "name": "title", "selector": "h2", "transforms": ["normalizeWhitespace"], "required": true },
      { "name": "price", "selector": "[itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
      { "name": "url", "selector": "h2 a[href]", "extract": "attributeValue", "attributeName": "href", "transforms": ["toAbsoluteUrl"] }
    ]
  }
}

The second version has no generated class and no position, reads the price from a machine-readable attribute, excludes ads explicitly and marks the fields that prove the page was parsed as required. That last part matters later, when something still breaks.

Fallback selectors while a site migrates

Sites rarely switch layouts in one step. Old and new templates often run side by side for weeks, on different pages or for different visitors. A single selector then works on half the pages.

An extractor field can take a list of selectors with combineStrategy: "first". The engine tries them in order and the first one that yields a non-empty value wins. “Non-empty” is judged on the value, not on the element: a selector whose element exists but has empty text, or lacks the attribute, falls through to the next one.

{
  "name": "title",
  "selector": ["[data-testid=\"product-title\"]", "main h1"],
  "combineStrategy": "first",
  "transforms": ["normalizeWhitespace"]
}

Two rules apply. A selector array without combineStrategy returns null, so always set it. And a comma string such as "h1, h2" is not a fallback list: it is one selector that matches either, in document order. When the old template is gone, remove the dead selector so the definition says what the page actually is.

Selectors the extraction engine does not support

ExtractHub extracts on a streaming HTML parser, not in a browser. That lets plain-HTML pages skip the browser entirely, but it limits the selector syntax. Supported: tags, .class, #id, attribute selectors ([attr], ="v", ^=, $=, *=, ~=), compounds, descendant and child combinators, :nth-child(), :first-child, :nth-of-type(), :first-of-type, :not() with a simple selector, and comma lists.

Not supported: +, ~, :has(), :is(), :where(), :last-child, :last-of-type, :nth-last-child(), :only-child, :contains(), pseudo-elements and XPath. The validate_extractor_config tool of the MCP server reports a selector the engine cannot parse as an error, and the extractor builder skill’s bundled checker flags the same list before anything is uploaded.

The usual reason to reach for + or :has() is a value that only follows a label, as in <dt>Color</dt><dd>Red</dd>. Two options without them:

  • Position among same-tag siblings: #specs dl > dd:nth-of-type(2). Simple, but positions shift when a row is missing on some pages.
  • Read every row as a pair: a list of { label, value } objects over the dt / dd or tr rows, and let your code map labels to fields. More robust when rows vary from page to page.

When the page has JSON, skip selectors

The most robust selector is often the one that points at a <script> block. Many sites ship the same data as JSON-LD, __NEXT_DATA__ or a window state object, and that JSON follows a data model rather than a design. A redesign can replace every class on the page and leave offers.price exactly where it was. The full method is in scrape embedded JSON. Cross-check the JSON against the visible page once; some sites leave stale values in it.

Catching the redesign that still gets through

No selector strategy is perfect. The goal is to break less often and to know at once when it happens.

  • Mark the fields that prove the page was parsed as required. An empty required top-level field fails the request with required_fields_missing (HTTP 422) instead of returning a record full of null. Mark the main list required too: a required sub-field whose selector breaks drops every item, and an optional list would then come back as a quiet [].
  • Use health checks. Public extractors are checked on a schedule, private ones when you turn checks on. A released version is probed on its sample page and judged on its required fields. How the probes decide between broken and degraded is in scraper monitoring.
  • Ship the fix as a patch. A selector change that keeps the output schema is a patch release, so callers pinned to @you/slug@1 get it without touching their code. Why that works is in versioned web scrapers.

A checklist for review, whether you write the selectors or an agent does:

  1. No generated class names, unless scoped and accepted as maintenance.
  2. No positional chains longer than one step.
  3. Every single-value selector scoped to the content block.
  4. Values from attributes (datetime, content, href, data-*) where the page has them.
  5. List selectors that exclude ads and placeholders, with the item count checked.
  6. One or two required fields per page type, and the main list required.
  7. Validated on a second page of the same type before release.

Want to see how others solved a site? The public extractor catalog shows each extractor’s output fields, and you can call a released version instead of writing your own.

FAQ

What is the most stable CSS selector for web scraping?

Hooks that exist for reasons other than styling: an id, a data-testid or other data-* attribute, an itemprop, or attributes such as rel, name and aria-label. Semantic tags inside a scope (main h1, article time[datetime]) come next; generated class names come last.

Why does my scraper break when the site changes its CSS?

Usually because a selector depends on a generated class name, on a long positional chain, or on the first match in the document. A new build renames the classes, a new wrapper shifts the positions, or a new block above the content becomes the first match.

Should I scrape display text or attributes?

Attributes, where the page has them. A datetime, content or href attribute is written for programs, while display text such as "2 days ago" or "Next" changes with language, A/B tests and copy edits.

How do I add a fallback selector for when a site changes its layout?

In an ExtractHub extractor, give the field's selector as an array with combineStrategy "first": the first selector that yields a non-empty value wins, so an element that exists but is empty falls through to the next one.

Does :has() work in web scraping selectors?

Not in an ExtractHub extractor. Extraction runs on a streaming HTML parser, not in a browser, so +, ~, :has(), :is(), :where(), :last-child and similar selectors are not supported. Use :nth-of-type on same-tag siblings, or read label and value pairs as a list.

All posts