Blocked Web Source Recovery

An empirical answer to a question this wiki kept guessing at: when a source fetch fails, is that a tooling problem we can engineer around, or is the content actually unavailable? Mostly the latter. Measured 2026-08-11 against the failed records from the 2026-06-30 Zotero ingest.

Why this was tested

The Zotero ingest pipeline fetches page bodies with a plain HTTP client and a custom user agent. During the 2026-06-30 bulk ingest this left 104 raw records carrying fetch_status: failed. The obvious hypothesis was that a real browser would fix most of them — many sites reject non-browser user agents or require JavaScript to render, and a headless Chrome driven through the DevTools Protocol defeats both.

That hypothesis is largely wrong, and the shape of why it is wrong determines where recovery effort is worth spending.

The testable corpus is smaller than the failure count

Of the 104 failed records, only 33 have HTTP(S) source URLs at all. The other 71 carry zotero://select/... deep links, which address entries in the local Zotero database rather than anything on the web. No fetch strategy can succeed against them; their web URLs were never extracted from Zotero in the first place.

This is the single most actionable finding, and it is not a web-access finding: 68% of the wiki's "failed fetches" are an upstream data-extraction defect, not blocked content. Fixing the ingest pipeline's URL resolution would shrink the failure set far more than any amount of browser engineering.

What browser rendering actually recovered

All 33 HTTP URLs were tested with a real Chrome instance, sequentially, with no authentication, cookies, CAPTCHA solving, paywall bypass, or robots.txt evasion.

Outcome	Count	Share
Recovered — full usable content	4	12.1%
Partial — incomplete content	2	6.1%
Still blocked	27	81.8%

The four recoveries were sondrabarrett.com, flextegrity.com, an Artsy artwork page for 4D Time World, and a CJ Fearnley blog post on the objective/subjective distinction — the last returning 373,383 characters, by far the most substantial single recovery.

Taxonomy of what blocks the rest

Blocking type	Count	Share	Recoverable by rendering?
dead or removed	14	42.4%	No — content is gone
bot detection	10	30.3%	No — challenges unsolved
soft paywall or meter	2	6.1%	No — blocked at network level
hard paywall	2	6.1%	No — subscription required
JavaScript required, unresolved	1	3.0%	Partially — needs multi-step interaction
none (recovered)	4	12.1%	Yes

Dead or removed (14) is the largest category and the most final: DNS failures, expired certificates, 404s and 410s, parked domains, and redirects to nowhere. Affected hosts include bobwb.tripod.com, thenewpress.com, na2ure.org, stoutbooks.com, loc.gov, and — notably — cjfearnley.com, whose blog survived while the main domain did not.

Bot detection (10) covers Cloudflare challenges, CAPTCHAs, and outright access-denied responses: JSTOR, MIT Press (both mitpress.mit.edu and direct.mit.edu), the New York Times, World Scientific, the Norman Foster Foundation, Fast Company. Driving a real browser is not the same as passing a bot check, and the test deliberately did not attempt to defeat one.

Paywalls (4 combined) are legitimate access restrictions rather than technical failures. The two DOI-resolved academic URLs return citation metadata of a few hundred characters and nothing more.

The operational conclusion

Browser rendering is worth adding, but as a narrow retry pass, not a replacement strategy. It addresses roughly one URL in eight — the slice blocked purely by user-agent filtering. The remaining failures divide into content that no longer exists (route to the Wayback Machine, a separate research task), content behind deliberate access controls (mark permanently failed and stop retrying), and the large zotero:// cohort (fix upstream in the ingest pipeline).

Treating all 104 failures as one problem is the mistake this measurement corrects. They are three problems with three different owners, and only the smallest of them is about browsers.

Caveats

This is a single point-in-time measurement with n=33 and no repeated trials, which is why confidence here is medium and volatility is warm rather than cold. Live web blocking is not a stable property: dead links stay dead, but bot-detection posture and paywall metering both change without notice, so the recovery rate should be re-measured rather than assumed if it is ever load-bearing.

One bookkeeping inconsistency in the source note is worth flagging: its blocking table reports non_html_or_binary as 0, while its accompanying text assigns that classification to one of the two partial results. The discrepancy does not affect the totals or any conclusion above.

Sources

Agent Browser Web Content Blocking Research — session research note, 33-URL test with per-URL classification (TODO-20)