Repository navigation
lab-ops-hub: container crawler's 20s timeout is per-socket-read, not a deadline — a slow-drip target hangs the single-threaded crawler indefinitely #532
Description
Problem
The Lab Ops Hub container crawler's 20-second timeout is a per-socket-operation timeout, not a deadline — a slow-drip target defeats it indefinitely, unlike the Worker fallback's absolute 20s deadline. Worse, the container serves requests on a single thread, so one dripping crawl wedges the whole crawler service.
Evidence on current main (a8a3bf7):
pkgs/lab-ops-hub/container/app.py:363-364—with _OPENER.open(req, timeout=20) as res: raw = res.read(2_000_000). In Python'surllib,timeoutbounds each blocking socket operation (connect, eachread), not the request as a whole. A server that sends one byte every <20s keeps every individual read inside the timeout while the total crawl runs unbounded — up to 2,000,000 drips before the byte cap is reached.pkgs/lab-ops-hub/container/app.py(last line) —HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()is the plain single-threaded server: while one/crawlhandler is stuck in the drip read, every other request (including theGET /health check) queues behind it.- The Worker fallback does not have this hole:
pkgs/lab-ops-hub/src/lib.ts:248-256(crawlFetcher, lab-ops-hub: Worker-fallback crawler has no fetch timeout — a slow target hangs the crawl indefinitely #500/lab-ops-hub: the #500 crawl-timeout fix has no regression test — crawlFallback's fetcher is inline in the untestable Worker entry #524) attachesAbortSignal.timeout(20_000)to the fetch, and the signal stays attached to the response, so the header wait and thereadResponseTextCappedbody read share one absolute 20s deadline per hop. - lab-ops-hub: Worker-fallback crawler has no fetch timeout — a slow target hangs the crawl indefinitely #500's issue body cites the container's
timeout=20as the model the Worker fix followed — but the two 20s values have different semantics, so the container path never actually had the guarantee lab-ops-hub: Worker-fallback crawler has no fetch timeout — a slow target hangs the crawl indefinitely #500 attributed to it.
Reproduction (verified this run)
Local drip server (headers, then 1 byte/second) read with exactly the container's call shape but timeout=3:
with urllib.request.urlopen(url, timeout=3) as res:
raw = res.read(2_000_000)The read completed after ~15s (15 bytes at 1 byte/s) despite the 3s timeout — each individual socket read finished within 3s, so no timeout ever fired. With the production timeout=20 and a 19s drip interval, a crawl could in principle run for months of wall-clock time; in practice it runs until something external kills it, blocking the single-threaded server the whole time.
Why it matters
POST /api/crawl targets are caller-supplied public URLs (validated, but a validated public server can still be slow or malicious). One drip crawl:
- holds the container's only request thread indefinitely — the health endpoint and all later crawls stall;
- defeats the
sleepAfter=2midle-kill economics the README relies on (the container is busy, not idle, and burns container minutes); - makes the Worker's 120s container-dispatch timeout fire while the container keeps crawling in the background, so retries pile onto an already-wedged service.
Acceptance criteria
- The container crawl enforces a total deadline for a whole
/crawlfetch (headers + body, per redirect hop or overall — document which), comparable to the Worker fallback's 20sAbortSignal.timeoutsemantics. - A regression test in
pkgs/lab-ops-hub/test/test_container_validate.py(or a sibling suite run bynpm test) drives a drip/slow server (or a fake transport with the same per-read behaviour) and shows the crawl aborting at the deadline instead of completing. - The fix does not weaken any existing validation:
npm testinpkgs/lab-ops-hubpasses (both TS and Python suites). - Optional but welcome: note the single-threaded
HTTPServercoupling in the fix or file it as a follow-up — aThreadingHTTPServeralone does not fix the unbounded crawl, so it is not a substitute for the deadline.
Suggested fix
Track a monotonic deadline in the crawl path: e.g. wrap the response read in a loop with res.read1/read(65536) chunks accumulating to the existing 2,000,000-byte cap, checking time.monotonic() against a start timestamp before each chunk and aborting past the deadline (chunked reads also match the Worker's capped-reader shape). Redirect hops go through the same opener, so decide and document whether the deadline is per hop (matching the Worker) or for the whole crawl.
Activity
dezren39 commentedon Oct 26, 2026
Intake review
Verified still live on current main
14dd66b6(2026-10-26, fleet monitor). Inpkgs/lab-ops-hub/container/app.py, the crawl still opens with_OPENER.open(req, timeout=20)(:1230 — the per-socket-operation timeout the body describes; the line has moved since filing as the file grew) and the container still serves with the plain single-threadedHTTPServer(...).serve_forever()(:1320) — noThreadingHTTPServer, no absolute deadline. The slow-drip hole and the one-stuck-crawl-wedges-everything failure mode are both intact. The issue arrived with a full acceptance-criteria section and a verified reproduction, so this pass is labels + this comment only.Related: #500 / #524 — the Worker fallback fix (
AbortSignal.timeout(20_000)inpkgs/lab-ops-hub/src/lib.ts) whose absolute-deadline semantics are the model to mirror; the body shows the container'stimeout=20was cited as the model for that fix while meaning something weaker.Risks / suggested approach: effort/s: give the container crawl one absolute deadline shared across connect, headers and body read (mirroring the Worker's per-hop 20s), and consider
ThreadingHTTPServerso one stuck crawl cannot wedge the health endpoint — the body's acceptance criteria cover both the deadline and a drip-server regression test. The security shape (a reachable slow endpoint as a one-request DoS of the whole crawler service) is why this is priority/normal rather than low despite the small diff.dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
558cc1d23e1d2d90df2a0d36189b5e585f12f99b.pkgs/lab-ops-hub/container/app.py:1230—response = _OPENER.open(req, timeout=20)(per-socket-operation timeout, unchanged since filing; the line moved from :363 as the file grew).pkgs/lab-ops-hub/container/app.py:1237—return status, res.read(2_000_000)with no total deadline or chunk loop.pkgs/lab-ops-hub/container/app.py:1320— plain single-threadedHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()(import at :22 isHTTPServeronly, noThreadingHTTPServer).urlopen(url, timeout=3)thenres.read(2_000_000)) completed in 8.1s with 8 bytes — no timeout ever fired, confirming the timeout bounds each socket read, not the fetch.repo-health:v1:fix:developing-today/code:issue:532): monotonic total deadline in_fetch_crawl, chunked reads to the existing 2 MB cap, per-hop vs whole-crawl documented, drip regression test in the lab-ops-hub Python suite,npm testinpkgs/lab-ops-hubgreen. ThreadingHTTPServer explicitly excluded as a substitute.queue/queuedfor the follow-up.1 remaining item
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
repo-health-code-pilot; issue revalidated open on 2026-11-01.dezren39 commentedon Nov 1, 2026
🤖 Agent note · handed off
npm testpassed (TypeScript 53/53, Python 64/64); focusedTestFetchCrawlpassed 11/11.skipping(neutral), with no pending CI checks.dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
pr_opened.npm testpassed (TS 53/53, Python 64/64); focused fetch tests passed 11/11. PR is open, merge state clean, and no CI checks are pending (CodeRabbit skipped).