lab-ops-hub: container validator returns non-ASCII path/query/userinfo verbatim — fetcher raises UnicodeEncodeError on URLs both validators accept #615

Problem

The container validator (pkgs/lab-ops-hub/container/app.py, validate_crawl_url) returns non-ASCII characters (U+0080 and above) verbatim in the path, query and userinfo of an accepted URL, and the container fetcher then dies on them:

• WHATWG URL parsing (the Worker's validateCrawlUrl in src/lib.ts, via new URL) UTF-8 percent-encodes every non-ASCII character in the userinfo, path, query and fragment, returning a clean ASCII URL.
• The container validator accepts the same spellings but returns the raw string. _fetch_crawl hands it to urllib/http.client, which encodes the request line (and headers) as ASCII and raises UnicodeEncodeError before any request is sent — so do_POST surfaces a 502 for a URL both validators accepted, instead of the crawl result the Worker path produces.

This is the judge-vs-fetch family of #611/#614, for the character class #614 does not cover: #614's scope and acceptance criteria are the C0 controls (U+0000–U+001F), space and DEL in the returned URL. Non-ASCII characters are a different class with a different remedy (UTF-8 percent-encoding, not trimming/control-encoding), and any ordinary non-English crawl target hits it — a pasted https://example.com/café or a search URL with a non-English query is enough. Verdicts agree everywhere; this is acceptance-side only, no SSRF direction (blocked hosts stay refused — the host judging is untouched, and a non-ASCII host itself is fetched fine via IDNA, as the existing café.com parity cases show).

Evidence (reproduced on current main 3c31fcd, 2026-10-22)

Differential through both real validators (container validate_crawl_url vs Worker validateCrawlUrl under Bun):

'http://example.com/café'        container returns 'http://example.com/café'        Worker returns 'http://example.com/caf%C3%A9'
'http://example.com/日本語'       container returns 'http://example.com/日本語'       Worker returns 'http://example.com/%E6%97%A5%E6%9C%AC%E8%AA%9E'
'http://example.com/?q=café'     container returns 'http://example.com/?q=café'     Worker returns 'http://example.com/?q=caf%C3%A9'
'http://example.com/?q=😀'       container returns 'http://example.com/?q=😀'       Worker returns 'http://example.com/?q=%F0%9F%98%80'
'http://user:café@example.com/'  container returns 'http://user:café@example.com/'  Worker returns 'http://user:caf%C3%A9@example.com/'
'http://café@example.com/'       container returns 'http://café@example.com/'       Worker returns 'http://caf%C3%A9@example.com/'

Fetcher consequence, verified with the container's own _fetch_crawl against a local HTTP server (host swapped to the test server, non-ASCII positions preserved):

'http://<srv>/café'      -> UnicodeEncodeError: 'ascii' codec can't encode character 'é' (no request sent)
'http://<srv>/日本語'     -> UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-7 (no request sent)
'http://<srv>/?q=café'   -> UnicodeEncodeError (same)
'http://café@<srv>/x'    -> UnicodeEncodeError (userinfo)

Controls that fetch fine through the container and are out of scope: a non-ASCII fragment (/x#café — the fragment is never sent), and the ASCII characters WHATWG also encodes (", <, >, backtick, ^, {, } in the path) — urllib sends those verbatim and the server answers, so they are returned-form differences only, not fetcher failures.

Why it matters

validate_crawl_url's contract is judge-what-you-fetch. Any crawl target whose path or query carries a non-ASCII character — non-English page names, search queries, generated links, redirect Locations built from user data (the redirect handler validates through the same function) — is accepted by the container validator and then fails the fetch with a 502, while the Worker fallback fetches the WHATWG-normalized form successfully. The two crawlers disagree on the outcome for a large class of ordinary public URLs.

Acceptance criteria

• For every input both validators accept, the URL validate_crawl_url returns is pure ASCII outside the host — non-ASCII characters in the userinfo, path, query and fragment are UTF-8 percent-encoded the way WHATWG serializes them (the host keeps its current spelling: it is judged in mapped form and fetched via IDNA, per the existing design).
• No double-encoding: existing %XX sequences in those components pass through byte-identical, and verdict parity is preserved (no currently-accepted public target changes verdict; blocked hosts stay refused).
• Python tests pin the headline cases (path café, CJK path, query, emoji query, non-ASCII password and username in userinfo) with the exact returned URLs, and the TS suite pins the Worker's returned form for the same inputs.
• A differential run over a non-ASCII battery (path/query/fragment/userinfo positions) shows verdict parity and ASCII-only returned URLs from the container validator outside the host, and _fetch_crawl successfully fetches the returned form for a local target carrying non-ASCII path/query.