lab-ops-hub: container validator accepts IDNA-invalid hosts the Worker refuses — unassigned codepoints and bidi-violating mixed labels #583

Summary

The container crawl validator (pkgs/lab-ops-hub/container/app.py, validate_crawl_url) certifies hosts that are not valid IDNA/UTS-46 domain names at all — and that the Worker's validator refuses outright — in (at least) two whole classes the hand-maintained checks cannot see:

1. Unassigned codepoints. A label containing a codepoint Unicode has not assigned (e.g. U+0378, U+0380, U+05EB) is accepted and passed on as a public DNS name.
2. Bidi-rule violations. A label mixing left-to-right Latin with right-to-left Hebrew/Arabic characters (e.g. exאample.com, localאhost) is accepted, even though IDNA's bidi requirements forbid it — Python's own stdlib IDNA codec raises Violation of BIDI requirement 2 on exactly this input, but the validator never calls it.

The Worker side needs no code to reject these: new URL() / WHATWG domain-to-ASCII (UTS-46, CheckBidi + disallowed/unassigned checks) throws, and validateCrawlUrl (pkgs/lab-ops-hub/src/lib.ts) returns url is not a valid URL. So the two validators that are supposed to hold the same crawl-target policy (#441 defence in depth: the component that actually fetches re-validates independently) disagree on thousands of spellings.

Evidence (verified 2026-10-22 on main 59ba8cf1)

Differential probe: every codepoint U+0080–U+32FF inserted into ex<cp>ample.com and local<cp>host, plus script-digit IP spellings and ASCII structural insertions — 26,040 hosts run through both real validators (Python validate_crawl_url imported from container/app.py; TS validateCrawlUrl imported from src/lib.ts):

• 3,174 divergences, all in one direction (container accepts / Worker refuses); 0 in the reverse direction.
• By first non-ASCII codepoint: ~2,070 unassigned, ~1,001 RTL bidi (AL/R), plus smaller pockets (e.g. U+0601, U+066B, U+115F, enclosed digits like U+2478).

Headline cases:

Root cause, container/app.py:

• validate_crawl_url (line 62) never performs an IDNA validity check on the mapped host. Its only IDNA-adjacent step is the length check at line 190, which measures "xn--" + label.encode("punycode") — raw punycode encoding succeeds for any string, valid IDN or not.
• The invalid-codepoint defence is the hand-enumerated _IDNA_DISALLOWED frozenset (line 55, from lab-ops-hub: container validator still allows IDNA-disallowed Unicode hosts (space separators, bidi/format controls, U+FFFD) the Worker validator refuses #516: ~30 codepoints). A denylist cannot cover unassigned ranges (thousands of codepoints, growing/shrinking per Unicode version) or contextual rules like bidi, which are properties of the whole label, not of single codepoints.
• Even the obvious quick fix is insufficient on its own: "exאample".encode("idna") raises the bidi error (verified), but "\u0378".encode("idna") happily returns b"xn--zva" — the stdlib IDNA-2003 codec accepts unassigned codepoints that UTS-46 marks disallowed. Parity needs a check that matches WHATWG behaviour for both classes, not just a codec call.

Consequences, same shapes this series has fixed before (#502, #507, #516):

• The container certifies a non-name as a public DNS name; for bidi-violating spellings the failure then surfaces deep inside the fetcher (the resolver's IDNA step raises) as a 502 crawl failure, instead of the validator's up-front refusal — and the Worker fallback never sees the target the same way, so behaviour depends on which crawler runs.
• Defence in depth is weakened asymmetrically: any future check layered on "the validator only passes real DNS names" (blocked-suffix, length, punycode A-label handling — cf. open lab-ops-hub: container validator still accepts the A-label (punycode) spelling of Arabic-Indic-digit hosts that PR #575 refused in raw form — including xn--9hbcp.xn--8hb.xn--8hb.xn--9hb, the encoded 127.0.0.1 #580) is reasoning about a host form this gap lets through unjudged in IDNA terms.

Acceptance criteria

• validate_crawl_url refuses hosts whose labels are not valid UTS-46/IDNA domain labels — covering at minimum the unassigned-codepoint and bidi-violation classes above — with the same accept/refuse verdict as the Worker's validateCrawlUrl on a differential battery (the 26,040-host probe above, or an equivalent committed test corpus).
• Pure RTL labels, ordinary IDN (münchen.de), and every currently-accepted public host in the existing suites stay accepted; CONTEXTJ handling stays as tracked separately in lab-ops-hub: container validator blanket-refuses CONTEXTJ-valid ZWNJ/ZWJ hosts (virama and Arabic-script joining contexts) the Worker validator accepts #531 (this issue is about spellings the Worker refuses).
• New tests in test/test_container_validate.py pin the headline cases (mixed LTR+RTL label refused, unassigned codepoint refused, pure RTL label accepted), and the TS suite gains the same cases if it does not already pin them.
• No regression in the existing Python (32 tests) or TS suites.

Notes

• Single-repo: validate_crawl_url / _IDNA_DISALLOWED exist only in developing-today/code (grep across the opencode / openchamber / nix clones: no copies). Not an mcpx set.
• Related, distinct: lab-ops-hub: container validator still accepts the A-label (punycode) spelling of Arabic-Indic-digit hosts that PR #575 refused in raw form — including xn--9hbcp.xn--8hb.xn--8hb.xn--9hb, the encoded 127.0.0.1 #580 (A-label/punycode spelling of Arabic-Indic digits — an encoded form of an already-covered refusal), lab-ops-hub: container validator blanket-refuses CONTEXTJ-valid ZWNJ/ZWJ hosts (virama and Arabic-script joining contexts) the Worker validator accepts #531 (opposite direction: container over-refuses CONTEXTJ-valid ZWNJ/ZWJ). This issue is the remaining under-refusal: whole IDNA-validity classes neither validator check enumerates.