Skip to content

lab-ops-hub: container validator blanket-refuses Arabic-Indic digits (U+0660–U+0669) inside RTL labels the Worker accepts — #574's range overrides #595's contextual AN rule #605

Closed

Description

@dezren39

Summary

The container crawl validator (pkgs/lab-ops-hub/container/app.py) refuses every host containing an Arabic-Indic digit (U+0660–U+0669), in any label context. The Worker's validator (pkgs/lab-ops-hub/src/lib.ts, WHATWG domain-to-ASCII) does not: it accepts those digits inside a genuine right-to-left label — the context RFC 5893 allows bidi class AN (Arabic number) in — and refuses them outside it.

This is the pocket the #595 fix (PR #604) measured and deliberately left untouched: its differential found 10 opposite-direction diffs where the container refuses U+0660–U+0669 inside an RTL label while the Worker accepts that context under both Bun and Node. It is also an internal inconsistency: #595 taught _judge_idna_label the contextual AN rule for the other AN digits (the astral Hanifi Rohingya / Rumi digits, and the U+066B/U+066C separators), and that helper already judges U+0660–U+0669 correctly — the older blanket range from #574 simply overrides its verdict in the one context where the digit is valid. The neighbouring Extended Arabic-Indic digits (U+06F0–U+06F9, class EN) are already accepted in the same RTL labels, so an Arabic label spelled with ١ is refused while the same label spelled with ۱ is accepted.

Evidence (verified 2026-10-22 on main 1695ee52)

Two blanket sites, both unconditional on label context:

  • container/app.py:293 — validate_crawl_url's forbidden-character check: or 0x0660 <= ord(c) <= 0x0669, applied to the whole host.
  • container/app.py:413 — the same range in _judge_a_label, applied to a decoded A-label before _judge_idna_label is called.

The contextual judgement already exists and already agrees with the Worker on every neighbouring case:

  • container/app.py:506 — if "AN" in bidis and (has_ltr or not has_rtl): refuse ("contains an Arabic number outside a right-to-left label").
  • container/app.py:511 — if "AN" in bidis and "EN" in bidis: refuse ("mixes Arabic and European numbers").

Differential on current main, driving the real container validate_crawl_url (Python) and the real Worker validateCrawlUrl (Bun; Node agrees on every row):

URL Container Worker
http://ا١.com/ (Arabic letter + U+0661) REFUSE "forbidden character" ALLOW → xn--mgb0j.com
http://مرحبا١.com/ REFUSE "forbidden character" ALLOW → xn--mgbbop3hzh.com
http://א١.com/ (Hebrew letter + U+0661) REFUSE "forbidden character" ALLOW → xn--4db40a.com
http://ا١ا.com/ (AN mid-label, RTL) REFUSE "forbidden character" ALLOW → xn--mgba7o.com
http://xn--mgb0j.com/ (A-label of row 1) REFUSE "decodes to a label with a forbidden character" ALLOW
http://xn--mgbbop3hzh.com/ (A-label of row 2) REFUSE (same) ALLOW
http://١.com/ (AN alone, no RTL char) REFUSE (via :506, correct) REFUSE
http://١٢٧.٠.٠.١/ (127.0.0.1 spelling, #574) REFUSE (via :506, correct) REFUSE
http://ex١ample.com/ (LTR label) REFUSE (via :506, correct) REFUSE
http://א١1.com/ (RTL + AN + EN mix) REFUSE (via :511, correct) REFUSE
http://١ا.com/ (RTL label not starting R/AL) REFUSE (correct) REFUSE
http://ا۱.com/ (Extended Arabic-Indic U+06F1 control) ALLOW ALLOW

Note the refused-for-the-right-reason rows: they are refused by _judge_idna_label (which runs before the host-level check for raw non-ASCII labels) — the blanket range is not what keeps #574's spellings out; in raw form it only ever fires on the RTL-context rows the judge has already accepted.

Why it matters

Same Worker/container parity class as #516/#526/#529/#531, in the fail-closed direction: the Worker accepts these public IDN hosts and dispatches the crawl, the container then refuses them, so legitimate crawls of Arabic/Hebrew-script domains whose names contain an ordinary Arabic-Indic digit fail at crawl time. The container is also the sole judge for redirect targets (its validating redirect handler validates raw Location values with no Worker normalisation in between), so a redirect to such a host 502s via the container while the Worker fallback resolves it. Arabic-Indic digits are the everyday digits of Arabic orthography — this is not an exotic spelling class.

Suggested fix

Delete the 0x0660 <= ord(c) <= 0x0669 arm from both forbidden-character checks (app.py:293 and app.py:413) and update the two comments that claim WHATWG rejects any host containing one. U+0660–U+0669 are bidi class AN, so after the deletion they are judged exactly like the other AN characters: _judge_idna_label refuses them in an LTR label, in a label with no R/AL character, and mixed with EN, and accepts them inside a genuine RTL label. For the A-label path the same applies — _judge_a_label already ends by calling _judge_idna_label(decoded, host).

Acceptance criteria

Single-repo: the validator pair exists only in this repo (validate_crawl_url / validateCrawlUrl grep-clean in the opencode / openchamber / nix clones). Dedup: #574/#580 (closed) are the source of the blanket range and scoped their refusals to the non-RTL spellings above; #595 (closed) covers the astral AN digits and explicitly left this pocket; open #531 is the ZWNJ/ZWJ CONTEXTJ blanket (different characters, different check), #588 is A-label decoded-label validity classes the container accepts, #596 is the U+0345 mapping — none covers a refused RTL-context AN digit.

Activity

  1. added a commit that references this issue on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions