Repository navigation
Description
Summary
PR #590 (fixes #583, commit 6bad7e15, merged as ead788e1) judges raw hosts in their mapped form in validate_crawl_url (pkgs/lab-ops-hub/container/app.py, current main ead788e1): the host is NFKC-normalized and lowercased before _judge_idna_label applies its bidi check. But the container's "mapped form" is NFKC, and UTS-46 mapping is not NFKC: U+0345 (COMBINING GREEK YPOGEGRAMMENI) is mapped by UTS-46 to U+03B9 (ι, bidi class L), while unicodedata.normalize("NFKC", "\u0345") returns it unchanged (its UnicodeData decomposition field is empty; the mapping lives in the special-casing data — Python's own "\u0345".casefold() does produce ι). The container therefore judges U+0345 as what it is before UTS-46 mapping — a combining mark, bidi NSM, legal mid-label in any context — while the Worker maps it to a Greek letter first and then applies bidi validity to the result. Inside an RTL label that flips the verdict: the mapped label is R + L + R, a bidi-rule violation.
PR #590's own comment in _judge_idna_label names U+0345 as a known residual, but only in the opposite direction ("U+0345, which UTS-46 maps to a letter, reads as a leading mark here" — the container over-refusing it at label start). The under-refusal inside RTL labels is not documented anywhere.
Evidence (verified 2026-10-22 on current main ead788e1, the merge commit of PR #590 itself)
Python validate_crawl_url imported from container/app.py at main vs the Worker's validateCrawlUrl from src/lib.ts under Bun; raw-form rows cross-checked with Node's new URL, which agrees with Bun on every raw case:
| URL | Container | Worker |
|---|---|---|
http://א\u0345א.com/ (U+0345 between two Hebrew letters) |
accepts | refuses — url is not a valid URL |
http://א\u0345.com/ (Hebrew letter + U+0345) |
accepts | refuses |
http://xn--jua51kba.com/ (A-label spelling of the first row) |
accepts | refuses under Bun (Node's URL parser accepts this A-label spelling — engine-dependent, secondary) |
Controls: http://ex\u0345ample.com/, http://α\u0345.com/ (LTR/Greek contexts) |
accepts | accepts |
Documented reverse residual: http://\u0345abc.com/ |
refuses (leading mark) | accepts (maps to ιabc) — unchanged, already disclosed in PR #590's comment |
Scope: this comes from a differential battery inserting every BMP codepoint U+0080–U+FFFF between two Hebrew letters (א<cp>א.com) through both real validators — U+0345 and U+FE12 (#593, a different root cause) are the only container-accept / Worker-refuse divergences in that position, and U+0345 produces none in Latin-mid, leading or trailing positions (leading is the documented reverse case above). A targeted scan confirms the mechanism is the mapping gap: "א\u0345א" after the container's NFKC+lowercase steps is still ['0x5d0', '0x345', '0x5d0'], so _judge_idna_label sees bidis [R, NSM, R] — no L, no mixing, trailing rule skips the NSM — while the Worker's mapped form is [R, L, R].
Why it matters
Same shape as #583: the two validators that are supposed to hold the same crawl-target policy (#441 defence in depth — the container is the component that actually fetches) disagree, here because the container's approximation of UTS-46 mapping (NFKC + lower()) silently diverges from UTS-46 mapping for a codepoint PR #590 itself flagged — and only the harmless direction of that divergence was documented. Any future codepoint whose UTS-46 mapping changes its bidi class or category relative to NFKC lands in the same hole; U+0345 is the one the current batteries can see.
Acceptance criteria
validate_crawl_urlrefuses a host whose UTS-46-mapped label violates the bidi rule because of the U+0345 → U+03B9 mapping — at minimumhttp://א\u0345א.com/andhttp://א\u0345.com/are refused with the same verdict as the Worker'svalidateCrawlUrl, in raw form and (under the repo's Bun differential) in the A-label spellingxn--jua51kba.- The LTR/Greek controls (
ex\u0345ample.com,α\u0345.com) stay accepted; the documented leading-position over-refusal may stay as-is (Worker-accepts residual) or be closed by the same mapping fix, but must not silently change without a test pinning whichever behaviour is chosen. - New tests in
pkgs/lab-ops-hub/test/test_container_validate.py(and the TS pin intest/lib.test.ts) cover U+0345 in an RTL label (refused) and in a Greek label (accepted). - No regression in the existing Python or TS suites.
Single-repo: validate_crawl_url / _judge_idna_label exist only in developing-today/code (git-grep verified in the opencode / openchamber / nix clones). Not an mcpx set. Deduped: searches for 0345 / ypogegrammeni return nothing; #593 (U+FE12 separator mapping) and #595 (missing AN rule) are different root causes in the same helper from the same PR; coordinate with #588 for the A-label spelling.
Activity