Skip to content

lab-ops-hub: container validator allows percent-encoded hosts (%6cocalhost, 127.0.0.1%09) and IPv6 zone ids that the Worker validator refuses #505

Closed

Description

@dezren39

Summary

The lab-ops-hub container crawler's validate_crawl_url accepts any host containing a % as "a public DNS name" (or, for IPv6, as a scoped address), while the Worker's WHATWG-based validateCrawlUrl percent-decodes the host first and refuses every one of these spellings. The container validator is the defence-in-depth check on the component that actually fetches, so it must not wave through host spellings the Worker validator rejects.

Evidence

Verified on current main eaf371c9 by driving both real validators over the same inputs (TS via node --experimental-strip-types importing src/lib.ts, Python by importing container/app.py): 19 divergences, all Python ALLOW / TS BLOCK, e.g.

Input Python container TS Worker
http://%6cocalhost/ ALLOW BLOCK (decodes to localhost)
http://localhost%00/ ALLOW BLOCK
http://127.0.0.1%09/ ALLOW BLOCK
http://8.8.8.8%09/ (also %0a, %0d, %20, %00, %zz, %25, %23) ALLOW BLOCK
http://example.com%09/, http://exa%09mple.com/, http://foo%bar.com/, http://example.com%/, http://example.com%2/ ALLOW BLOCK
http://[2001:4860:4860::8888%25eth0]/ ALLOW BLOCK

Root cause:

  • pkgs/lab-ops-hub/container/app.py — validate_crawl_url takes urlparse(...).hostname verbatim. A host containing % is not an IP literal (ipaddress rejects it), does not match _inet_aton_ipv4's character class ([0-9a-fA-FxX.]+), and its final label is usually not numeric, so it falls through to return raw # a public DNS name. For bracketed IPv6, ipaddress accepts a zone/scope id (2001:4860:4860::8888%eth0), so a scoped global address is allowed.
  • WHATWG URL parsing percent-decodes the host before validation: %6cocalhost becomes localhost (refused as a local name), most other % spellings are invalid hosts outright, and zone ids are not accepted — new URL("http://127.0.0.1%09/") throws.

Note the fetcher angle: with glibc's resolver these literal % hosts do not resolve today (getaddrinfo("127.0.0.1%09") fails), so this is a validator-parity / defence-in-depth hole rather than a demonstrated fetch of a loopback target — the same class as #502/#504, where the container side must refuse what the Worker side refuses instead of reclassifying the spelling as a public DNS name. Percent-decoding resolvers and non-glibc stacks are exactly what the independent re-validation exists for.

Controls that already agree and must not change: http://%65xample.com/ ALLOW/ALLOW (decodes to a public name), http://example%2ecom/ ALLOW/ALLOW, http://127%2e0.0.1/ BLOCK/BLOCK, http://%31%32%37.0.0.1/ BLOCK/BLOCK, http://[fe80::1%25eth0]/ BLOCK/BLOCK.

Why it matters

Two validators implementing one crawl policy (#441) keep drifting on host spellings (#466, #468, #472, #474, #476, #502, #504). A blanket rule for % closes the whole spelling family at once instead of one decoded form at a time.

Acceptance criteria

  • validate_crawl_url in container/app.py refuses any URL whose host contains a raw % (percent-encoded host spellings and IPv6 zone ids), matching the Worker validator's verdicts on the table above.
  • New cases in test/test_container_validate.py cover at least: %6cocalhost, localhost%00, 127.0.0.1%09, 8.8.8.8%zz, a malformed trailing %, and [2001:4860:4860::8888%25eth0].
  • Existing suites still pass: python3 -m unittest discover -s test (Python) and npm test (TS) in pkgs/lab-ops-hub.
  • The %65xample.com / example%2ecom control spellings are considered explicitly: if the fix refuses all % hosts, the TS side's ALLOW for those two decodable-public spellings becomes the remaining (fail-closed, container-stricter) divergence and should be noted in the PR.

Activity

  1. added a commit that references this issue on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions