Repository navigation
Description
Summary
PR #508 (fix for #507) added a whitespace check to the lab-ops-hub container crawler's validate_crawl_url (pkgs/lab-ops-hub/container/app.py:63 on main d4809cff):
if any(ord(c) <= 0x20 for c in host):
raise ValueError(f"refusing to crawl host with whitespace {host!r}")Its comment says "A space or control character in the host is not a DNS name", but ord(c) <= 0x20 covers only the C0 controls + space. The remaining control characters and the rest of the WHATWG forbidden-host set still fall through and are certified as "a public DNS name", while the Worker's validateCrawlUrl (pkgs/lab-ops-hub/src/lib.ts:129) refuses every one of them outright:
| Host form | TS Worker | Python container |
|---|---|---|
http://exam\x7fple.com/ (DEL) |
BLOCK ("url is not a valid URL") | ALLOW |
http://exam\x80ple.com/ … \x9f (C1 controls) |
BLOCK | ALLOW |
http://exam\xa0ple.com/ (NBSP) |
BLOCK | ALLOW |
http://exam<ple.com/, exam>ple, exam^ple, exam|ple |
BLOCK | ALLOW |
Controls, identical on both sides (must not change): ordinary DNS names, http://exam_ple.com/, and valid IDN hosts (e.g. examéple.com) are ALLOW/ALLOW — so the fix must not blanket-ban non-ASCII.
Evidence
- PR: fix(lab-ops-hub): container validator refuses space hosts and IPvFuture literals #508, commit
ebbeb8bca08895de704c720526b7842235590249, merged asd4809cffa77d9fb4695262445802a197ab1b6b4e. - Verified on current main
d4809cff(gitorigin/mainandgh apiagree) by driving both real validators over the same battery: every printable ASCII character plus 0x7F–0xA0, é and a combining char, each embedded ashttp://exam<char>ple.com/(TS via bun importingsrc/lib.ts, Python by importingcontainer/app.py). The complete set of divergences was exactly:<,>,^,|, 0x7F, 0x80–0x9F, 0xA0. - The DEL case reproduces lab-ops-hub: container validator accepts space hosts and IPvFuture literals the Worker validator refuses #507's exact lazy-failure shape: Python's own fetcher rejects it deep inside
http.client, whose disallowed-character check is[\\x00-\\x20\\x7f]— i.e.http.client.HTTPConnection("exam\\x7fple.com").putrequest(...)raisesInvalidURL: URL can't contain control characters— but the validator's new check stops one codepoint short of the fetcher's own set. The< > ^ |hosts are WHATWG forbidden host codepoints; no DNS resolver accepts them either, so those crawls also die below the validator instead of at it.
Why it matters
The container validator is the defence-in-depth re-check on the component that actually fetches (#441), and the parity premise of #468/#502/#505/#507 is that where the two validators disagree, the effective policy is the weaker one. PR #508 closed the space/C0 part of this class but left the validator's stated rule ("space or control character") under-implemented: a whole tail of not-a-DNS-name hosts is still certified as public DNS names, pushing the failure into the fetcher/resolver as a low-level error and leaving the guard's verdict meaningless for them.
Cross-repo note: this validator pair lives only in developing-today/code (git grep for validateCrawlUrl/validate_crawl_url in the opencode, openchamber and dezren39/nix clones finds no twin), so this is a single-repo issue.
Acceptance criteria
validate_crawl_urlrefuses a host containing DEL (0x7F), a C1 control (0x80–0x9F), NBSP (0xA0), or any of< > ^ |, up front with a clear validator error.- Controls unchanged: ordinary DNS names,
exam_ple.com, valid IDN hosts, valid IPv6 literals and public targets are still accepted on both sides. - A regression test in
pkgs/lab-ops-hub/test/test_container_validate.pycovers each refused class, and fails with onlycontainer/app.pyreverted (ablation). - The Python and TS validators agree on the full character battery above.
Activity
Merge pull request #515 from developing-today/sweep/510-forbidden-hos…