Repository navigation
Description
Problem
The container crawl validator (pkgs/lab-ops-hub/container/app.py, validate_crawl_url, line 62) parses the URL authority with urllib.parse.urlparse, which treats \ as an ordinary netloc character (part of the userinfo). The Worker's WHATWG URL parser treats \ as a path delimiter for special schemes (http/https), i.e. it ends the authority. The two validators therefore disagree about which host a backslash-spelled URL names — in both directions.
Verified this run against current main (container validator imported directly; WHATWG via Node's new URL, inputs built with chr(92)/String.fromCharCode(92) so each URL contains exactly one backslash):
| Input | WHATWG host (Worker judges) | urlparse host (container judges) | Result |
|---|---|---|---|
http://127.0.0.1\@example.com/ |
127.0.0.1 |
example.com |
Worker refuses (non-public IPv4); container accepts |
http://localhost\@example.com/ |
localhost |
example.com |
Worker refuses (blocked host); container accepts |
http://example.com\@127.0.0.1/ |
example.com |
127.0.0.1 |
Worker accepts; container refuses (non-public) |
http:\\example.com/ |
example.com |
(none) | Worker accepts; container refuses "url has no host" |
http:/\example.com/ |
example.com |
(none) | Worker accepts; container refuses "url has no host" |
Why it matters
- The container validator is the only judge for redirect targets:
container/app.py:401-407re-validates every redirectLocationwithvalidate_crawl_urlalone, and the container's own crawl endpoint validates with it independently (app.py:498). A validator that reads127.0.0.1\@example.comasexample.comcertifies a URL whose WHATWG spelling names the loopback host — any WHATWG-semantics consumer of that certified URL (a future fetcher change, a proxy, a browser-side re-crawl) connects to a host this validator never judged. This is the same judge-the-host-that-gets-fetched invariant the lab-ops-hub: container validator allows percent-encoded hosts (%6cocalhost, 127.0.0.1%09) and IPv6 zone ids that the Worker validator refuses #505/lab-ops-hub: container validator blanket-refuses percent-encoded hosts the Worker validator decodes and accepts — public spellings (%65xample.com, 8%2e8%2e8%2e8) diverge, including as redirect targets #529/lab-ops-hub: PR #533 decode-and-judge bypass — percent-encoded fullwidth delimiters NFKC-map into host delimiters after the decode check, so the container fetches unjudged private hosts (localhost / 127.0.0.1 / 192.168.0.1) #567 fixes established. - In the other direction, Worker-accepted crawl targets (
http://example.com\@127.0.0.1/,http:\\example.com/) are refused by the container, so the container fallback errors on URLs the primary path considers valid — an availability divergence of the same family as lab-ops-hub: redirect policy diverges between crawlers — Worker fallback caps at 5 hops and follows any 3xx, container follows 10 hops and only 301/302/303/307/308 #571.
Note the percent-encoded form %5C is data, not a delimiter, in both parsers and must stay judged as today; only a raw \ is structural.
Acceptance criteria
validate_crawl_urljudges the host the WHATWG parser names for raw-backslash spellings:http://127.0.0.1\@example.com/andhttp://localhost\@example.com/are refused as non-public/blocked;http://example.com\@127.0.0.1/is accepted asexample.com(and the URL actually fetched namesexample.com— the returned/validated form must not hand the fetcher a spelling whose urlparse host differs from the judged host).http:\\example.com/andhttp:/\example.com/are handled the WHATWG way (hostexample.com) or explicitly refused with a test pinning the choice — but the two validators must not silently disagree.- New tests in
pkgs/lab-ops-hub/test/test_container_validate.pycover the table above (Python suite stays green; TS suite untouched unless the fix changes Worker behavior). - Differential probe: a battery of backslash spellings (in host, userinfo, and authority-prefix positions) yields the same accept/refuse verdict and the same judged host in both validators.
Activity
lab-ops-hub: ruff format the #577 tests
Merge pull request #579 from developing-today/creator/577-backslash-a…
Merge pull request #621 from developing-today/creator/620-backslash-s…