Skip to content

lab-ops-hub: container validator returns IP literals un-canonicalized — inet_aton spellings and expanded IPv6 kept verbatim, Worker returns the canonical address #637

Closed

Description

@dezren39

Finding

The container validator judges IP-literal hosts correctly (the
inet_aton spellings of private addresses are refused, #468/#502), but
it returns accepted IP literals in the spelling the caller typed,
while the Worker validator returns the canonical form. For the same
accepted target the two crawlers therefore record — and hand to their
fetchers and logs — different URL strings:

  • IPv4 short / integer / hex / octal spellings are returned verbatim:
    1.2.3, 1.2.256, 134744072, 0x8080808, 010.0.0.010,
    8.8.2056, 8.16777215, 1.2.3.04, 0x8.0x8.0x8.0x8.
  • IPv6 literals are returned un-compressed and un-lowercased:
    [2001:4860:4860:0000:0000:0000:0000:8888].

validate_crawl_url (pkgs/lab-ops-hub/container/app.py:78) parses
the literal for judging (ipaddress.ip_address / _inet_aton_ipv4,
app.py:364-366) but its success returns (app.py:440-442, and the
DNS-name return at :400-402) hand back
_url_with_encoded_components(raw) — the raw spelling — unless the
host was percent-decoded. The Worker validator
(pkgs/lab-ops-hub/src/lib.ts:130) returns u.toString(), the
WHATWG-canonical form (canonical dotted quad; compressed lowercase
IPv6).

This is the IP-literal remainder of the returned-form divergence:
#633 covers dot segments, host case and default ports in the returned
URL, but not the IP-literal spellings themselves; #468/#502 fixed
which addresses are judged private, not the form that is returned.

Evidence

Verified this run on main 5a84351c, driving both real validators
(Python validate_crawl_url; Worker's validateCrawlUrl under Bun).
Allow/refuse agreed on every probe (all private inet_aton spellings
refused by both); only the returned form differs:

Input Container returns Worker returns
http://1.2.3/ http://1.2.3/ http://1.2.0.3/
http://1.2.256/ http://1.2.256/ http://1.2.1.0/
http://134744072/ http://134744072/ http://8.8.8.8/
http://0x8080808/ http://0x8080808/ http://8.8.8.8/
http://010.0.0.010/ http://010.0.0.010/ http://8.0.0.8/
http://8.8.2056/ http://8.8.2056/ http://8.8.8.8/
http://8.16777215/ http://8.16777215/ http://8.255.255.255/
http://1.2.3.04/ http://1.2.3.04/ http://1.2.3.4/
http://0x8.0x8.0x8.0x8/ http://0x8.0x8.0x8.0x8/ http://8.8.8.8/
http://[2001:4860:4860:0000:0000:0000:0000:8888]/ verbatim http://[2001:4860:4860::8888]/

Fetcher-side, the container's resolver does reach the same address
(socket.getaddrinfo: 1.2.256 → 1.2.1.0, 134744072 → 8.8.8.8,
8.8.2056 → 8.8.8.8), so this is not an SSRF bypass. The harm is
the standing one for this validator pair: the judged, recorded and
fetched spellings diverge between the two crawlers. Anything keyed on
the URL string (crawl history, dedup, audit logs, a downstream
consumer that parses the returned URL with a strict dotted-quad
parser instead of inet_aton semantics) sees 134744072 where the
Worker recorded 8.8.8.8 — a host string that was never the judged
canonical address.

Suggested fix

In the IP-literal success path of validate_crawl_url, return the
URL with the host replaced by the canonical form of the address that
was judged — str(ip) for IPv4, [compressed-lowercase] for IPv6 —
via the existing _url_with_host helper (app.py:674), the same way
the percent-decoded path already substitutes the judged host.
DNS-name returns are out of scope (host case there is #633).

Acceptance criteria

  • For every accepted IP-literal input, the container validator
    returns the same URL string as the Worker's validateCrawlUrl,
    including the table above.
  • Allow/refuse verdicts are unchanged on the full existing probe
    corpus (private inet_aton spellings still refused).
  • New cases in pkgs/lab-ops-hub/test/test_container_validate.py
    pin the canonical returns; the Python suite passes and the TS
    suite is untouched.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions