Repository navigation
Description
Summary
The container crawl validator (pkgs/lab-ops-hub/container/app.py) refuses every host containing an Arabic-Indic digit (U+0660–U+0669), in any label context. The Worker's validator (pkgs/lab-ops-hub/src/lib.ts, WHATWG domain-to-ASCII) does not: it accepts those digits inside a genuine right-to-left label — the context RFC 5893 allows bidi class AN (Arabic number) in — and refuses them outside it.
This is the pocket the #595 fix (PR #604) measured and deliberately left untouched: its differential found 10 opposite-direction diffs where the container refuses U+0660–U+0669 inside an RTL label while the Worker accepts that context under both Bun and Node. It is also an internal inconsistency: #595 taught _judge_idna_label the contextual AN rule for the other AN digits (the astral Hanifi Rohingya / Rumi digits, and the U+066B/U+066C separators), and that helper already judges U+0660–U+0669 correctly — the older blanket range from #574 simply overrides its verdict in the one context where the digit is valid. The neighbouring Extended Arabic-Indic digits (U+06F0–U+06F9, class EN) are already accepted in the same RTL labels, so an Arabic label spelled with ١ is refused while the same label spelled with ۱ is accepted.
Evidence (verified 2026-10-22 on main 1695ee52)
Two blanket sites, both unconditional on label context:
container/app.py:293—validate_crawl_url's forbidden-character check:or 0x0660 <= ord(c) <= 0x0669, applied to the whole host.container/app.py:413— the same range in_judge_a_label, applied to a decoded A-label before_judge_idna_labelis called.
The contextual judgement already exists and already agrees with the Worker on every neighbouring case:
container/app.py:506—if "AN" in bidis and (has_ltr or not has_rtl): refuse("contains an Arabic number outside a right-to-left label").container/app.py:511—if "AN" in bidis and "EN" in bidis: refuse("mixes Arabic and European numbers").
Differential on current main, driving the real container validate_crawl_url (Python) and the real Worker validateCrawlUrl (Bun; Node agrees on every row):
| URL | Container | Worker |
|---|---|---|
http://ا١.com/ (Arabic letter + U+0661) |
REFUSE "forbidden character" | ALLOW → xn--mgb0j.com |
http://مرحبا١.com/ |
REFUSE "forbidden character" | ALLOW → xn--mgbbop3hzh.com |
http://א١.com/ (Hebrew letter + U+0661) |
REFUSE "forbidden character" | ALLOW → xn--4db40a.com |
http://ا١ا.com/ (AN mid-label, RTL) |
REFUSE "forbidden character" | ALLOW → xn--mgba7o.com |
http://xn--mgb0j.com/ (A-label of row 1) |
REFUSE "decodes to a label with a forbidden character" | ALLOW |
http://xn--mgbbop3hzh.com/ (A-label of row 2) |
REFUSE (same) | ALLOW |
http://١.com/ (AN alone, no RTL char) |
REFUSE (via :506, correct) | REFUSE |
http://١٢٧.٠.٠.١/ (127.0.0.1 spelling, #574) |
REFUSE (via :506, correct) | REFUSE |
http://ex١ample.com/ (LTR label) |
REFUSE (via :506, correct) | REFUSE |
http://א١1.com/ (RTL + AN + EN mix) |
REFUSE (via :511, correct) | REFUSE |
http://١ا.com/ (RTL label not starting R/AL) |
REFUSE (correct) | REFUSE |
http://ا۱.com/ (Extended Arabic-Indic U+06F1 control) |
ALLOW | ALLOW |
Note the refused-for-the-right-reason rows: they are refused by _judge_idna_label (which runs before the host-level check for raw non-ASCII labels) — the blanket range is not what keeps #574's spellings out; in raw form it only ever fires on the RTL-context rows the judge has already accepted.
Why it matters
Same Worker/container parity class as #516/#526/#529/#531, in the fail-closed direction: the Worker accepts these public IDN hosts and dispatches the crawl, the container then refuses them, so legitimate crawls of Arabic/Hebrew-script domains whose names contain an ordinary Arabic-Indic digit fail at crawl time. The container is also the sole judge for redirect targets (its validating redirect handler validates raw Location values with no Worker normalisation in between), so a redirect to such a host 502s via the container while the Worker fallback resolves it. Arabic-Indic digits are the everyday digits of Arabic orthography — this is not an exotic spelling class.
Suggested fix
Delete the 0x0660 <= ord(c) <= 0x0669 arm from both forbidden-character checks (app.py:293 and app.py:413) and update the two comments that claim WHATWG rejects any host containing one. U+0660–U+0669 are bidi class AN, so after the deletion they are judged exactly like the other AN characters: _judge_idna_label refuses them in an LTR label, in a label with no R/AL character, and mixed with EN, and accepts them inside a genuine RTL label. For the A-label path the same applies — _judge_a_label already ends by calling _judge_idna_label(decoded, host).
Acceptance criteria
- Raw RTL spellings are accepted, matching the Worker:
http://ا١.com/,http://مرحبا١.com/,http://א١.com/,http://ا١ا.com/. - Their A-label spellings are accepted:
http://xn--mgb0j.com/,http://xn--mgbbop3hzh.com/,http://xn--4db40a.com/. - Every lab-ops-hub: container validator accepts hosts containing Arabic-Indic digits (U+0660–U+0669) that the Worker validator rejects outright — including the ١٢٧.٠.٠.١ spelling of 127.0.0.1 #574/lab-ops-hub: container validator still accepts the A-label (punycode) spelling of Arabic-Indic-digit hosts that PR #575 refused in raw form — including xn--9hbcp.xn--8hb.xn--8hb.xn--9hb, the encoded 127.0.0.1 #580 refusal still refuses:
١٢٧.٠.٠.١,١٢٣.com,a١.com,example١٢٣.com,münchen١.com, the percent-encoded spelling, and the A-label spellingsxn--9hbcp.…,xn--a-bqc,xn--example-h3jjk,xn--mnchen-3ya761i— now via the AN context rule rather than the blanket range. - AN+EN mixing in an RTL label still refuses (
א١1.com). - New tests in
pkgs/lab-ops-hub/test/test_container_validate.pypin the accepted raw + A-label RTL spellings (the TS suite needs no change — the Worker already accepts them; a parity pin there is welcome). - No regression in the existing Python or TS suites.
Single-repo: the validator pair exists only in this repo (validate_crawl_url / validateCrawlUrl grep-clean in the opencode / openchamber / nix clones). Dedup: #574/#580 (closed) are the source of the blanket range and scoped their refusals to the non-RTL spellings above; #595 (closed) covers the astral AN digits and explicitly left this pocket; open #531 is the ZWNJ/ZWJ CONTEXTJ blanket (different characters, different check), #588 is A-label decoded-label validity classes the container accepts, #596 is the U+0345 mapping — none covers a refused RTL-context AN digit.
Activity
Merge pull request #606 from developing-today/creator/605-arabic-indi…