Repository navigation
lab-ops-hub: redirect policy diverges between crawlers — Worker fallback caps at 5 hops and follows any 3xx, container follows 10 hops and only 301/302/303/307/308 #571
Description
Summary
The two lab-ops-hub crawlers disagree on redirect policy, so the same public URL can succeed via one crawler and fail via the other — and the Worker fallback (which exists to stand in when the container is unavailable) does not reproduce the container's behaviour.
Evidence (verified 2026-10-22, on main 7a9ac873)
Redirect-count limit
- Worker fallback:
fetchFollowingValidatedRedirectsdefaults tomaxRedirects = 5(pkgs/lab-ops-hub/src/lib.ts:204, error at lib.ts:216-218). - Container:
_ValidatingRedirectHandler(pkgs/lab-ops-hub/container/app.py:389-396) subclasses urllib'sHTTPRedirectHandlerand inherits its defaults:max_redirections = 10,max_repeats = 4.
Redirect status codes
- Worker fallback follows any 3xx response that carries a
Locationheader (lib.ts:res.status >= 300 && res.status < 400). - urllib's handler only treats 301, 302, 303, 307 and 308 as redirects; any other 3xx raises
HTTPError, which the container'sdo_POSTsurfaces as a 502.
Reproduction (both sides run this run)
- A 7-hop chain of 302s between public URLs, ending in 200:
- Worker fallback (real
fetchFollowingValidatedRedirectsunderbun testwith a scripted fetcher):Error: too many redirects (more than 5). - Container redirect handler (urllib opener against a local server):
200.
- Worker fallback (real
- A single 300 Multiple Choices response with a
Locationpointing at a public 200 page:- Worker fallback: follows it, returns
200. - Container:
HTTPError 300→/crawlreturns 502.
- Worker fallback: follows it, returns
Why it matters
/api/crawl tries the container first and silently falls back to the Worker on any container error (src/index.ts crawlWithContainer / catch → crawlFallback), recording only source in the result. Redirect-heavy public pages (shorteners, AMP/print chains, locale negotiation) therefore crawl or fail depending on which backend happened to serve the request, and a 6–10-hop page that the container handles fine hard-fails whenever the container is cold or down — exactly when the fallback is supposed to be equivalent. This is the same class of Worker/container divergence fixed for URL validation (#460, #474, #481, #500, #529, #567) and extraction (#569); redirect policy is the remaining unaligned piece.
Acceptance criteria
- One redirect policy stated in one place both implementations follow: a single hop limit and a single set of followed status codes (suggested: align the Worker to the container's urllib semantics — 10 hops, codes 301/302/303/307/308 — or pick 5 deliberately and enforce it in the container handler too; either way the two must match, including a repeat/loop rule equivalent to urllib's
max_repeats). - The container handler sets its limits explicitly rather than relying on urllib defaults, so a Python/urllib default change cannot silently re-open the divergence.
- Tests on both sides pin the shared policy: a chain one hop over the limit fails identically, a chain at the limit succeeds identically, and a non-followed 3xx (e.g. 300/304 with
Location) is handled identically (TS tests inpkgs/lab-ops-hub/test/lib.test.ts, Python tests alongsidetest/test_container_validate.pyor a new container redirect test file). - Per-hop SSRF re-validation of every redirect target is unchanged on both sides (lab-ops-hub: Worker-fallback crawler follows redirects to non-public targets (no redirect validation, unlike the container) #460).
Activity
dezren39 commentedon Oct 26, 2026
Intake review
Still live (verified this run, fleet monitor): on current code main,
pkgs/lab-ops-hub/src/lib.ts:231still defaults the Worker fallback'sfetchFollowingValidatedRedirectstomaxRedirects = 5, while the container's_ValidatingRedirectHandlerinpkgs/lab-ops-hub/container/app.pystill inherits urllib's defaults (10 hops, only 301/302/303/307/308 treated as redirects). The divergence the issue reproduces — a 7-hop chain and a 300-with-Location — is unchanged.Related: the lab-ops-hub Worker/container divergence series: URL validation (#460, #474, #481, #500, #529, #567 line) and extraction (#569); #588 is the remaining A-label item from the same family. This is the last unaligned redirect-policy piece, per the issue body.
Notes / risks:
/api/crawlfalls back silently, so the divergence is user-visible only as inconsistent crawl success — the fix should state the single policy in one place both implementations reference, as the acceptance criteria require, rather than tweaking one constant. Aligning the Worker up to the container's 10 hops slightly widens fallback fetch behaviour; that is the container's already-shipping policy, not a new exposure.Suggested approach: align the Worker to the container's urllib semantics (10 hops; follow only 301/302/303/307/308, other 3xx surfaces as an error), and add the issue's two reproduction cases (7-hop 302 chain, single 300) as tests on both sides.
Labels normalized by the fleet monitor: kind/bug, source/pr-diff-review, status/needs-triage, priority/normal, effort/m, app/lab-ops-hub.
dezren39 commentedon Oct 26, 2026
Correction to the intake review above (line number only)
The fleet monitor's Intake review cited the Worker fallback default at
pkgs/lab-ops-hub/src/lib.ts:231. Fresh re-verification this run puts it at lib.ts:210 (maxRedirects = 5, with the limit error at :222-223). The substance of the review is unchanged: the Worker still defaults to 5 hops while the container inherits urllib's 10-hop policy, and the issue remains live. The wrong line number came from a stale tool read caught in this run's correction pass.dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
558cc1d23e1d2d90df2a0d36189b5e585f12f99b.pkgs/lab-ops-hub/src/lib.ts:210—maxRedirects = 5;:219follows any 3xx carrying aLocation(res.status < 300 || >= 400returns, everything else with a Location is followed); limit error at:222-223.pkgs/lab-ops-hub/container/app.py:1180—_ValidatingRedirectHandleroverrides onlyredirect_request(target validation + userinfo auth); a grep of the whole file finds nomax_redirections/max_repeatsset anywhere, so urllib's inherited defaults (10 hops / 4 repeats, codes 301/302/303/307/308 only) govern — exactly the divergence filed.max_repeats = 4. The container is the primary path and the fallback's job is to reproduce it; dropping the primary to 5 would break pages it crawls fine today. A non-followed 3xx classifies as a crawl failure on both sides (the container's 502 / lab-ops-hub: container crawler records redirect loops and over-limit chains as successful 302 crawls — PR #573's blanket HTTPError catch swallows urllib redirect errors #576 classification), not a returned result.repo-health:v1:fix:developing-today/code:issue:571): Worker limit/codes/repeat rule + container limits made explicit + shared-policy comments + both-sides tests (at-limit, over-limit, 300/304, loop),npm testinpkgs/lab-ops-hubgreen, per-hop SSRF re-validation unchanged. lab-ops-hub: redirect policy diverges between crawlers — Worker fallback caps at 5 hops and follows any 3xx, container follows 10 hops and only 301/302/303/307/308 #571 stays open; labels synced toqueue/queued.3 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
npm testpasses (56 TypeScript + 63 Python tests);git diff --checkpasses. CI has no applicable checks on this PR.Merge pull request #3772 from developing-today/queue/4839-571-redirec…