Repository navigation
Description
Finding
The container validator judges IP-literal hosts correctly (the
inet_aton spellings of private addresses are refused, #468/#502), but
it returns accepted IP literals in the spelling the caller typed,
while the Worker validator returns the canonical form. For the same
accepted target the two crawlers therefore record — and hand to their
fetchers and logs — different URL strings:
- IPv4 short / integer / hex / octal spellings are returned verbatim:
1.2.3,1.2.256,134744072,0x8080808,010.0.0.010,
8.8.2056,8.16777215,1.2.3.04,0x8.0x8.0x8.0x8. - IPv6 literals are returned un-compressed and un-lowercased:
[2001:4860:4860:0000:0000:0000:0000:8888].
validate_crawl_url (pkgs/lab-ops-hub/container/app.py:78) parses
the literal for judging (ipaddress.ip_address / _inet_aton_ipv4,
app.py:364-366) but its success returns (app.py:440-442, and the
DNS-name return at :400-402) hand back
_url_with_encoded_components(raw) — the raw spelling — unless the
host was percent-decoded. The Worker validator
(pkgs/lab-ops-hub/src/lib.ts:130) returns u.toString(), the
WHATWG-canonical form (canonical dotted quad; compressed lowercase
IPv6).
This is the IP-literal remainder of the returned-form divergence:
#633 covers dot segments, host case and default ports in the returned
URL, but not the IP-literal spellings themselves; #468/#502 fixed
which addresses are judged private, not the form that is returned.
Evidence
Verified this run on main 5a84351c, driving both real validators
(Python validate_crawl_url; Worker's validateCrawlUrl under Bun).
Allow/refuse agreed on every probe (all private inet_aton spellings
refused by both); only the returned form differs:
| Input | Container returns | Worker returns |
|---|---|---|
http://1.2.3/ |
http://1.2.3/ |
http://1.2.0.3/ |
http://1.2.256/ |
http://1.2.256/ |
http://1.2.1.0/ |
http://134744072/ |
http://134744072/ |
http://8.8.8.8/ |
http://0x8080808/ |
http://0x8080808/ |
http://8.8.8.8/ |
http://010.0.0.010/ |
http://010.0.0.010/ |
http://8.0.0.8/ |
http://8.8.2056/ |
http://8.8.2056/ |
http://8.8.8.8/ |
http://8.16777215/ |
http://8.16777215/ |
http://8.255.255.255/ |
http://1.2.3.04/ |
http://1.2.3.04/ |
http://1.2.3.4/ |
http://0x8.0x8.0x8.0x8/ |
http://0x8.0x8.0x8.0x8/ |
http://8.8.8.8/ |
http://[2001:4860:4860:0000:0000:0000:0000:8888]/ |
verbatim | http://[2001:4860:4860::8888]/ |
Fetcher-side, the container's resolver does reach the same address
(socket.getaddrinfo: 1.2.256 → 1.2.1.0, 134744072 → 8.8.8.8,
8.8.2056 → 8.8.8.8), so this is not an SSRF bypass. The harm is
the standing one for this validator pair: the judged, recorded and
fetched spellings diverge between the two crawlers. Anything keyed on
the URL string (crawl history, dedup, audit logs, a downstream
consumer that parses the returned URL with a strict dotted-quad
parser instead of inet_aton semantics) sees 134744072 where the
Worker recorded 8.8.8.8 — a host string that was never the judged
canonical address.
Suggested fix
In the IP-literal success path of validate_crawl_url, return the
URL with the host replaced by the canonical form of the address that
was judged — str(ip) for IPv4, [compressed-lowercase] for IPv6 —
via the existing _url_with_host helper (app.py:674), the same way
the percent-decoded path already substitutes the judged host.
DNS-name returns are out of scope (host case there is #633).
Acceptance criteria
- For every accepted IP-literal input, the container validator
returns the same URL string as the Worker'svalidateCrawlUrl,
including the table above. - Allow/refuse verdicts are unchanged on the full existing probe
corpus (private inet_aton spellings still refused). - New cases in
pkgs/lab-ops-hub/test/test_container_validate.py
pin the canonical returns; the Python suite passes and the TS
suite is untouched.
Activity
fix(lab-ops-hub): render canonical IPv6 from exploded hextets, not st…
fix(lab-ops-hub): derive canonical IPv6 hextets from the address inte…