lab-ops-hub: container validator still refuses tab/CR/LF-split slash-run authority spellings — WHATWG strips them before parsing, #587's collapse runs on the raw spelling #589

Summary

PR #587 (fixes #586) taught the container's validate_crawl_url to collapse the slash run after an http/https scheme colon to exactly // before urlparse. The collapse runs on the raw spelling — but WHATWG URL parsing first removes every ASCII tab (0x09), LF (0x0A) and CR (0x0D) from the input, and Python's urlparse does the same stripping internally when it later parses. A tab/CR/LF inside the delimiter run therefore splits the run for the PR's regex but not for either parser:

• http: TAB /example.com/x — the regex sees zero slashes before the tab, inserts // ahead of it, and urlparse's tab stripping then leaves http:///example.com/x (three slashes, empty netloc).
• http:/ TAB /example.com/x — the regex collapses only the first slash segment; stripping joins the second slash into a three-slash spelling.

The Worker's validateCrawlUrl (WHATWG new URL) accepts and normalizes all of these to http://example.com/x; the container refuses them as url has no host. This is the same over-refusal divergence shape as #586, one character class over: the run the regex collapses is not the run the parsers see.

Evidence (verified on current main 6bd0533, merge of PR #587; commit 1140d1d)

pkgs/lab-ops-hub/container/app.py:88-89:

if re.match(r"(?i)https?:", raw):
    raw = re.sub(r"(?i)^(https?:)/*", r"\g<1>//", raw)

Differential probe, container validate_crawl_url vs Worker validateCrawlUrl under Bun (the reference runtime for this pair, per #586):

Container REFUSES (url has no host), Worker ACCEPTS and normalizes to http://example.com/x:

• http: + TAB + /example.com/x
• http: + LF + //example.com/x (also CR)
• http:/ + TAB + /example.com/x (also LF, and CR+LF)
• https: + TAB + /example.com/

Blocked targets in these spellings are refused by both, but the container gives the wrong reason — https:/ + TAB + /127.0.0.1/ and http: + TAB + //127.0.0.1/x come back url has no host from the container vs non-public IPv4 from the Worker, unlike plain slash-run spellings after #587, which are refused for the right reason.

Redirect exposure is the #586 one: _ValidatingRedirectHandler validates raw Location values with this validator alone — a redirect to http:/ TAB /example.org/ is refused by the container (verified directly against the handler) but crawls through the Worker fallback, which resolves it with new URL(...).

Scope check (same probe, tab/CR/LF inserted at every position of http://example.com/x and http://127.0.0.1/x): the only verdict divergences are the delimiter-run positions above. Tabs elsewhere (inside the host, path, or directly after the colon with no following slash, e.g. http: TAB example.com/x) already agree, because urlparse strips them before judging. No spelling was found where the container accepts a host the Worker refuses in this family — over-refusal only, not an SSRF bypass.

Why it matters

The container re-validates independently as defence in depth, and for redirect hops it is the only validator. A page redirecting to a tab-split slash-run spelling of a public URL crawls fine via the Worker fallback but fails via the container (502, redirect refused as hostless), depending purely on which crawler path serves the request — the divergence class #586 set out to close for slash runs, left open for the tab/CR/LF characters WHATWG removes before the run is even read.

Acceptance criteria

• validate_crawl_url judges tab/CR/LF-split delimiter runs by the WHATWG host: strip ASCII tab/LF/CR from the raw input (matching WHATWG's pre-parse removal and urlparse's own stripping) before — or as part of — the lab-ops-hub: judge slash-run authority spellings by the WHATWG host (#586) #587 slash-run collapse, so the spellings above validate and are fetched as http(s)://example.com/x.
• Blocked targets in those spellings are refused for the right reason: https:/ TAB /127.0.0.1/, http: TAB //localhost/x refuse as non-public / local hosts, not as hostless URLs.
• No verdict changes for any other spelling: the existing Python and TypeScript suites stay green, and a differential probe against the Worker validator (under Bun) over slash-run spellings with tab/CR/LF inserted at every position of the scheme/delimiter area shows no remaining verdict differences.
• Regression tests in test/test_container_validate.py cover at least: one accepted public tab-split spelling, one accepted newline-split spelling, and one refused blocked tab-split spelling asserting the non-"no host" reason.