lab-ops-hub: container crawler records redirect loops and over-limit chains as successful 302 crawls — PR #573's blanket HTTPError catch swallows urllib redirect errors #576

Summary

PR #573 (fixing #572) made the lab-ops-hub container crawler treat urllib.error.HTTPError as a crawl result, so target 4xx/5xx pages are extracted instead of becoming a 502. The catch is blanket, though: urllib also raises HTTPError for redirect failures — redirect loops (max_repeats) and chains past max_redirections in HTTPRedirectHandler.http_error_302 raise HTTPError carrying the 3xx code. _fetch_crawl now converts those into a "successful" result, so a redirect-looping URL is recorded as a completed crawl with status_code: 302 and the redirect stub's body as its text — the crawl no longer fails, and the Worker fallback (which would surface its own too-many-redirects error) never runs.

Evidence

• PR: lab-ops-hub: container crawler returns error-status pages as results, not 502 #573 (merged, commit 4bb8903ca36519049e3bb81bb9383c0bbfaa0fb9, merge ef2f2efd)
• pkgs/lab-ops-hub/container/app.py:412-430 on current main (ef2f2efd): _fetch_crawl catches urllib.error.HTTPError around _OPENER.open(...) with no status-code check and returns (status, body) for whatever it carries.
• CPython urllib.request.HTTPRedirectHandler.http_error_302: when a redirect target was already visited >= max_repeats times or len(visited) >= max_redirections, it raises HTTPError(req.full_url, code, self.inf_msg + msg, headers, fp) — code is the 3xx (e.g. 302), not a 4xx/5xx.
• Handler.do_POST (app.py:493) trusts _fetch_crawl's return and answers HTTP 200 with status_code set, so crawlWithContainer in src/index.ts accepts it as a container result (no fallback, source: container recorded in D1).

Reproduction (verified 2026-10-22 on current main ef2f2efd)

Scratch harness (outside the repo, validation patched to identity only because the validating redirect handler refuses 127.0.0.1 targets by design; production redirect targets are public URLs, the urllib mechanics are identical):

• Self-redirect loop (/loop → 302 Location: /loop): _fetch_crawl returns RESULT status=302 body=b'looping' — before lab-ops-hub: container crawler returns error-status pages as results, not 502 #573 this HTTPError propagated and do_POST returned 502.
• 12-hop distinct 302 chain (over urllib's max_redirections = 10): returns RESULT status=302 body=b'hop' instead of raising.
• Controls behave correctly: 3-hop and 6-hop chains return 200, a plain 404 returns status=404 with its body (the lab-ops-hub: container crawler turns target 4xx/5xx into 502 — urllib HTTPError never extracted, Worker silently re-crawls and misreports source #572 fix working as intended), and the existing Python suite passes 28/28.

Why it matters

A redirect loop or runaway chain is a crawl failure, and both crawlers used to surface it as one (container: 502 → Worker fallback → the fallback's too many redirects error). Now the container silently manufactures a plausible-looking result from the redirect response itself: D1/R2 record a completed crawl whose title/text/links are extracted from a redirect stub page, with status_code: 302. Nothing downstream can tell this apart from a genuine 302 page crawl.

Related but distinct: #571 tracks aligning the two crawlers' redirect policy (hop limit, followed codes, non-followed 3xx handling). This issue is narrower and newer — #571 was filed against main 7a9ac873, where these errors still surfaced as a 502; #573 changed that failure mode into a silent success after #571 was written. Aligning the policy in #571 does not by itself require un-masking these errors, and un-masking them does not require settling the policy.

Acceptance criteria

• _fetch_crawl treats only genuine error-status responses (4xx/5xx) as crawl results; an HTTPError carrying a 3xx code (redirect loop, over-limit chain, or other redirect failure) propagates so do_POST returns 502 and the existing Worker-fallback path runs, as before lab-ops-hub: container crawler returns error-status pages as results, not 502 #573.
• New Python tests in pkgs/lab-ops-hub/test/test_container_validate.py (or a sibling file) pin: a redirect loop raises instead of returning a 302 result, and a chain past the redirect limit raises instead of returning a 302 result; the existing lab-ops-hub: container crawler turns target 4xx/5xx into 502 — urllib HTTPError never extracted, Worker silently re-crawls and misreports source #572 tests (404/500 returned as results, transport failure still raises) keep passing.
• The Python suite stays green (28 existing tests + the new ones).