How to detect when a website change breaks your scraper

Scraper monitoring that catches layout changes: required fields as the contract, scheduled probes, broken vs degraded, and why two failures make an alarm.

Published
Reading time
9 min read
By
ExtractHub team
  • scraper monitoring
  • health checks
  • web scraping
  • reliability
  • extractors
On this page

A scraper rarely crashes when the site it reads changes. The request still returns 200, the parser still runs, and the output is a list of objects whose title is null. Nothing throws, so nothing alerts, and you find out when someone downstream asks why the dashboard is empty. Scraper monitoring is the practice of catching that moment yourself: deciding what a correct result looks like, checking it continuously, and telling a broken layout apart from a bad day at the target site.

This post walks through how ExtractHub does it for extractors (hosted, versioned web scrapers): the required-field contract, the request-time failure, the scheduled health probe, how a probe is classified, and why the list of required fields should stay short. The rules are general enough to copy into your own scrapers.

Why scrapers fail silently

There are three ways a scraper’s output goes wrong, and only one of them is loud:

  1. The fetch fails. A 403, a timeout, a challenge page. Your HTTP client sees it, and so does your retry logic.
  2. The page is fine, the selectors are not. The site renamed a class or moved a block. The response is a normal 200 page, and the extractor returns empty values.
  3. The page is a different page. A product was delisted and the URL now redirects to a search page, or the site serves a consent wall with a 200 status. The selectors are fine; they are just looking at the wrong document.

Status codes only catch the first case. The other two need a check on the extracted data itself, which means you first have to say what “correct” data is.

The contract behind scraper monitoring: required fields

In an ExtractHub extractor, a field marked required: true must have a value on every page of its kind. This flag is what both request-time checks and health probes judge. An illustrative example for a product page (the selectors fit an imaginary site, not a real one):

{
  "fields": [
    { "name": "title", "selector": "h1", "extract": "textContent", "transforms": ["normalizeWhitespace"], "required": true },
    { "name": "sku", "selector": "[itemprop=\"sku\"][content]", "extract": "attributeValue", "attributeName": "content", "required": true },
    { "name": "price", "selector": "[itemprop=\"price\"][content]", "extract": "attributeValue", "attributeName": "content", "transforms": ["toNumber"] },
    { "name": "rating", "selector": "[data-rating]", "extract": "attributeValue", "attributeName": "data-rating", "transforms": ["toNumber"] }
  ]
}

title and sku prove that the parser found the product. rating is optional because a new product has no reviews, and null is the correct answer for it. Empty means null, an empty string or an empty list; 0 and false count as values.

On a list page, mark the main list itself required along with the one sub-field that identifies an item. A required sub-field works differently from a top-level one: an item missing it is dropped from the list instead of failing the request. If a sub-field selector breaks, every item is dropped and the list comes back empty, so a required main list is what catches it.

Signal one: the request that fails

The first signal comes for free with every call. When a top-level required field is empty on the fetched page, the extract request fails with 422 required_fields_missing instead of returning a half-empty result:

{
  "success": false,
  "error": "Required field missing on the fetched page: sku",
  "errors": [
    {
      "code": "required_fields_missing",
      "message": "Required field missing on the fetched page: sku",
      "fields": ["sku"],
      "hint": "The page may be a challenge page, a consent wall or a different layout; retry later or check a live sample."
    }
  ],
  "requestId": "..."
}

Your integration can treat this code as “do not store this row” and count it. The check runs after the page was fetched, so when the request went through the scraping proxy it is charged like any other fetch (the body then carries charged: true and the cost); on a direct request without the proxy it is free. In the Batch API and in pipelines the item or step is retried once before it fails. The API reference lists every error code.

Request-time failures have a blind spot, though: they only tell you about the URLs you happened to call. An extractor that runs once a week, or whose callers swallow errors, can be broken for days before anyone looks at a log.

Signal two: scheduled health probes

The second signal does not depend on traffic. A released version gets a health test: a URL plus expectations, probed on a schedule with the version’s own request options and extractor config.

  • The default test probes the version’s sample page, the page its release check ran on. When there is none, it uses the example URL of the release.
  • Public extractors are probed every 12 hours at the platform’s cost. When the version is healthy and real traffic in the last 24 hours shows at least 20 calls with a 95% success rate or better, the scheduled probe is skipped for one interval: the traffic already shows the fetch working. Traffic never clears a failure, though; a version that is not healthy is always probed.
  • Private extractors are probed only after the owner turns on “Run health checks” in Studio, at an interval of 1, 3, 6, 12 or 24 hours. Each probe is a normal scrape at the extractor’s price tier, charged to the owner (see pricing). Probes that the site blocked are not charged, and probes pause if the balance runs out.
  • More test URLs. The owner can add test URLs and tune each one: the share of items in which required fields must be filled (80% by default), the minimum number of items in a required list (1 by default), and up to 20 custom checks.

After fixing a test or a selector, “Run check now” in the Studio health panel queues a probe at once instead of waiting for the schedule. Each test can be run this way once every 5 minutes, with a cap of 20 manual runs per user per hour.

How a probe is classified

Each probe ends in one of five run statuses. The order of the checks matters: the classifier first rules out the site’s doing, then looks at the selectors.

Run statusWhen
passEvery expectation holds.
blockedThe target refused or timed out on every attempt (403, 429, 503 and similar), or fields came back empty on a detected block page or a page under 1,000 bytes.
errorA problem on our side or with the test itself: an unexpected status such as 404, an extraction crash, or nothing extracted from a page that is not the test URL’s page.
broken_selectorA plausible 2xx page where a required field is empty on every item, a required list is empty, or nothing at all was extracted.
degradedRequired fields filled in fewer items than the test’s fill rate, a required list shorter than the minimum, a type that drifted on a required field (a number field returning text), or a failed custom check.

The “not the test URL’s page” rule handles the third failure mode from the start of this post. When a probe extracts nothing, the classifier compares the final URL after redirects and the page’s <link rel="canonical"> with the test URL. If they point to another document, the run is an error with the reason test URL leads to another page (...): update the test URL. The selectors are not blamed for a product that no longer exists.

Why one failing probe is not an alarm

A single probe can fail for reasons that fix themselves: an A/B test variant, a half-deployed release on the site, a page that was briefly different. Raising an alarm on each one teaches people to ignore alarms. So the version status is a small state machine over the run history:

  • A pass makes the version healthy and resets the failure streak.
  • A degraded or broken_selector run adds one to the streak and shows the version as degraded.
  • The version becomes broken on a broken_selector run that brings the streak to two or more, so the earliest alarm is the second failing probe in a row. A degraded run alone never makes a version broken.
  • A broken version stays broken until a probe passes.
  • blocked never adds to the streak; the version shows blocked (or stays broken if it already was). Being blocked is the site’s choice, not a selector problem.
  • error changes nothing.

When a version has several tests, each test keeps its own streak, and the version shows the worst status among its active tests. One passing test cannot hide another test’s failures.

Why required fields must stay short

It is tempting to mark every field required, on the theory that more checks catch more breakage. In practice it does the opposite.

Optional fields never fail a probe; their fill rates are recorded for the owner and nothing else. A required field, on the other hand, fails real requests with 422 whenever it is empty. Mark rating required and every product without reviews becomes a failed (and possibly charged) request, plus a broken_selector probe if the test page happens to be one of them. Your monitoring then reports breakage that is really normal data, and the signal you wanted gets lost in it.

required changes are also a new major version, in both directions, because they change which requests succeed. Choosing the right fields before the first release saves a breaking change later.

The rule that works:

  1. Mark as required the one to three fields that prove the page was parsed: the title, an id, the main list.
  2. Leave optional anything a valid page may lack: ratings, discounts, sale prices, secondary images.
  3. To watch an optional field anyway, add a custom check to the health test instead of making it required. Custom checks take a field path such as products[].price, an operator (not_empty, gt, gte, lt, lte, eq, contains, matches), an optional value and a minimum pass rate (80% by default). A failed custom check makes the probe degraded without failing a single API request.

If an agent builds your extractors, the extractor builder skill follows the same rule when it sets required.

Where the health status shows up

  • Studio: the health panel lists the test URLs and recent checks with their status, HTTP status, item count and reason.
  • The public extractor page: every extractor page in the catalog shows the health of the latest version, and its “Versions and health” section shows it per version.
  • MCP: search_extractors returns health_status for each result and get_extractor returns it per major and per version, so an agent can check it before choosing an extractor. See the MCP server.

Versions report healthy, degraded, broken, blocked, or unknown before the first probe has run.

From broken to fixed

Once a version is broken, the fix is usually a selector change that keeps the output schema the same. Under semantic versioning that is a patch release, and every caller pinned to the major (@you/slug@1) receives it on their next call without changing any code. Run a check on the new version, and the first passing probe marks it healthy again.

This is why monitoring and versioning belong together. Monitoring tells you that the site changed; versioning lets the fix reach callers immediately while protecting them from changes they did not ask for. The versioning post covers the pin side in detail.

A scraper monitoring checklist

Whether or not you use ExtractHub, these rules carry over to any scraper:

  1. Define correct output as a short list of fields that must never be empty.
  2. Fail the request when one of them is empty, instead of storing a half-empty row.
  3. Probe a known page on a schedule, independent of real traffic.
  4. Rule out the site first: blocks, tiny pages and redirects to other documents are not selector failures.
  5. Alarm on a streak, not on a single bad run, and clear the alarm only on a pass.
  6. Keep optional fields optional and watch them with softer checks.
  7. Ship fixes as patches so callers get them without touching their code.

FAQ

How do I know if my web scraper is broken?

Probe a known page on a schedule and judge the result against fields that must always have a value. A 200 response proves nothing on its own: a changed layout usually returns a normal page with empty fields.

What is the difference between a degraded and a broken scraper?

In ExtractHub a probe is broken_selector when a required field is empty on every item, a required list is empty or nothing at all was extracted. It is degraded when required fields are only partly filled, a type drifted or a custom check failed. A version is marked broken after two failing probes in a row that end on a broken_selector run.

How often are scrapers health checked?

Public extractors are probed every 12 hours at the platform's cost. Private extractors are probed only when the owner turns checks on, at an interval from hourly to daily, and each check is charged to the owner at the extractor's price tier.

Should I mark every field as required?

No. Mark the one to three fields that prove the page was parsed, such as a title, an id or the main list. Fields a valid page may lack, like ratings or discounts, should stay optional; to watch one without failing requests, add a not_empty custom check to the health test.

All posts