Repository navigation
Description
Summary
The lab-ops-hub container crawler's validate_crawl_url accepts any host containing a % as "a public DNS name" (or, for IPv6, as a scoped address), while the Worker's WHATWG-based validateCrawlUrl percent-decodes the host first and refuses every one of these spellings. The container validator is the defence-in-depth check on the component that actually fetches, so it must not wave through host spellings the Worker validator rejects.
Evidence
Verified on current main eaf371c9 by driving both real validators over the same inputs (TS via node --experimental-strip-types importing src/lib.ts, Python by importing container/app.py): 19 divergences, all Python ALLOW / TS BLOCK, e.g.
| Input | Python container | TS Worker |
|---|---|---|
http://%6cocalhost/ |
ALLOW | BLOCK (decodes to localhost) |
http://localhost%00/ |
ALLOW | BLOCK |
http://127.0.0.1%09/ |
ALLOW | BLOCK |
http://8.8.8.8%09/ (also %0a, %0d, %20, %00, %zz, %25, %23) |
ALLOW | BLOCK |
http://example.com%09/, http://exa%09mple.com/, http://foo%bar.com/, http://example.com%/, http://example.com%2/ |
ALLOW | BLOCK |
http://[2001:4860:4860::8888%25eth0]/ |
ALLOW | BLOCK |
Root cause:
pkgs/lab-ops-hub/container/app.py—validate_crawl_urltakesurlparse(...).hostnameverbatim. A host containing%is not an IP literal (ipaddressrejects it), does not match_inet_aton_ipv4's character class ([0-9a-fA-FxX.]+), and its final label is usually not numeric, so it falls through toreturn raw # a public DNS name. For bracketed IPv6,ipaddressaccepts a zone/scope id (2001:4860:4860::8888%eth0), so a scoped global address is allowed.- WHATWG URL parsing percent-decodes the host before validation:
%6cocalhostbecomeslocalhost(refused as a local name), most other%spellings are invalid hosts outright, and zone ids are not accepted —new URL("http://127.0.0.1%09/")throws.
Note the fetcher angle: with glibc's resolver these literal % hosts do not resolve today (getaddrinfo("127.0.0.1%09") fails), so this is a validator-parity / defence-in-depth hole rather than a demonstrated fetch of a loopback target — the same class as #502/#504, where the container side must refuse what the Worker side refuses instead of reclassifying the spelling as a public DNS name. Percent-decoding resolvers and non-glibc stacks are exactly what the independent re-validation exists for.
Controls that already agree and must not change: http://%65xample.com/ ALLOW/ALLOW (decodes to a public name), http://example%2ecom/ ALLOW/ALLOW, http://127%2e0.0.1/ BLOCK/BLOCK, http://%31%32%37.0.0.1/ BLOCK/BLOCK, http://[fe80::1%25eth0]/ BLOCK/BLOCK.
Why it matters
Two validators implementing one crawl policy (#441) keep drifting on host spellings (#466, #468, #472, #474, #476, #502, #504). A blanket rule for % closes the whole spelling family at once instead of one decoded form at a time.
Acceptance criteria
validate_crawl_urlincontainer/app.pyrefuses any URL whose host contains a raw%(percent-encoded host spellings and IPv6 zone ids), matching the Worker validator's verdicts on the table above.- New cases in
test/test_container_validate.pycover at least:%6cocalhost,localhost%00,127.0.0.1%09,8.8.8.8%zz, a malformed trailing%, and[2001:4860:4860::8888%25eth0]. - Existing suites still pass:
python3 -m unittest discover -s test(Python) andnpm test(TS) inpkgs/lab-ops-hub. - The
%65xample.com/example%2ecomcontrol spellings are considered explicitly: if the fix refuses all%hosts, the TS side's ALLOW for those two decodable-public spellings becomes the remaining (fail-closed, container-stricter) divergence and should be noted in the PR.
Activity
Merge pull request #506 from developing-today/creator/505-container-p…