Repository navigation
lab-ops-hub: container validator returns the raw U-label host for IDN targets — Worker returns the punycode A-label form it judged #659
Description
Problem
The container validator (pkgs/lab-ops-hub/container/app.py, validate_crawl_url) judges a non-ASCII host in its IDNA-encoded form — it even builds the A-labels at app.py:286-292 ("xn--" + label.encode("punycode")) for the RFC 1035 length check — but it never substitutes those A-labels into the URL it returns. The returned URL keeps the raw U-label host. The Worker validator (pkgs/lab-ops-hub/src/lib.ts) returns the WHATWG-canonical punycode host (u.hostname), so for every internationalized host the two validators certify different URLs for the same input.
Evidence (verified 2026-10-23, both validators run on current main f1a8abf)
| Input | Container returns | Worker returns |
|---|---|---|
http://café.com/ |
http://café.com/ |
http://xn--caf-dma.com/ |
http://münchen.de/path |
http://münchen.de/path |
http://xn--mnchen-3ya.de/path |
http://bücher.example/ |
http://bücher.example/ |
http://xn--bcher-kva.example/ |
http://😀.com/ |
http://😀.com/ |
http://xn--e28h.com/ |
Repro: call validate_crawl_url("http://café.com/") in container/app.py and validateCrawlUrl("http://café.com/") in src/lib.ts and compare the returned URLs.
Why it matters
The returned URL is the artifact the rest of the system stores, logs, dedupes on, and fetches. Two validators that are supposed to be interchangeable implementations of one policy disagree on the canonical form of every IDN target: the same crawl submitted through the container and through the Worker produces different stored URLs (breaking dedupe/comparison), and the container's form depends on the fetcher re-encoding the host at fetch time instead of the validator returning the wire form it actually judged. This is the same class of returned-URL divergence as #652 (UTS-46 ignored codepoints in the returned host), but it covers all non-ASCII labels, not just the ignored-codepoint subset — #652's fix alone would not close it.
Acceptance criteria
- For any accepted non-ASCII host, the container validator returns the URL with the host in A-label (punycode) form, matching the Worker byte-for-byte (including the DNS root dot handling from lab-ops-hub: container validator drops the DNS root dot only for the percent-encoded spelling — returned URL differs from the Worker and from its own literal-dot form #647 and UTS-46 mapping/ignored-codepoint deletion from lab-ops-hub: PR #523's container length check measures raw punycode, not the UTS-46-mapped form — mapped IDN labels still diverge from the Worker in both directions #526/lab-ops-hub: container validator returns UTS-46 ignored codepoints in the host it judged without them — returned URL differs from the Worker, and U+180E cannot be IDNA-encoded at fetch time #652).
- ASCII hosts, IPv4/IPv6 literals, ports, userinfo, and path/query/fragment handling are unchanged.
- New tests in
pkgs/lab-ops-hub/test/test_container_validate.pypin the returned A-label form for at least: a Latin-1 IDN (café.com), a multi-label IDN (bücher.example), and an astral-plane label (😀.com→xn--e28h.com); a Python↔TS differential over IDN spellings shows no remaining returned-host differences.
Activity
dezren39 commentedon Oct 23, 2026
Context note from the filer: earlier sweep work (#633/#639 fixes) deliberately left the IDN U-label spelling in the container's returned host untouched ("IDN spelling kept per #615") while fixing ASCII host lowercasing and port normalization, so this divergence is known-but-untracked rather than newly discovered. This issue exists so the remaining decision is explicit: either canonicalize to the A-label form like the Worker (acceptance criteria above), or close this as wontfix with the rationale recorded — in which case the backlog should note the returned-host difference as a permanent, accepted exception to the byte-for-byte differential.
dezren39 commentedon Oct 26, 2026
Intake review
Still live (verified this run, fleet monitor):
pkgs/lab-ops-hub/container/app.pyon current main still builds the A-labels (encoded_labels, lines ~343–353) only for the RFC 1035 length judgement and never substitutes them into the returned URL — the return paths rebuild from the judged (U-label) host, and the code comments themselves flag this as the separate open item ("an IDN spelling, whose return form is the separate #659", line ~180; "ordinary IDN spellings keep their established return form", line ~279). The Worker'su.hostnamepunycode form still differs for every non-ASCII host.Related: #652 (UTS-46 ignored codepoints in the returned host — fixed, and its return-path rebuild pattern is the template here), #647 (root-dot return form), #526 (UTS-46 mapping). Same file, same returned-URL divergence class.
Notes / risks: the return-path rebuilds are heavily precedent-laden (#647/#652/#660 all shaped them); substituting A-labels must compose with the root-dot and ignored-codepoint return forms already pinned by tests, or those regress. Effort is m for that reason, not xs.
Suggested approach: in the non-IP return path, rebuild the host from the already-computed
encoded_labelswhen the host is non-ASCII, and pin the issue's four example hosts (café.com, münchen.de, bücher.example, 😀.com) plus the root-dot combination intest_container_validate.pyagainst the Worker's outputs.Labels normalized by the fleet monitor: kind/bug, status/needs-triage, priority/normal, effort/m, app/lab-ops-hub.
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done — triage verdict: CONFIRMED on current main.
Reproduced on the laptop at
origin/maincf1b243f8b72a102573492340623efc2dbf692b3, runningvalidate_crawl_urlfrompkgs/lab-ops-hub/container/app.pyextracted byte-for-byte from that HEAD:http://café.com/http://café.com/http://xn--caf-dma.com/http://münchen.de/pathhttp://münchen.de/pathhttp://xn--mnchen-3ya.de/pathhttp://bücher.example/http://bücher.example/http://xn--bcher-kva.example/http://😀.com/http://😀.com/http://xn--e28h.com/http://café.com./http://café.com./http://example.com/http://example.com/(unchanged, as expected)Mechanism at HEAD: the A-labels are already computed for the RFC 1035 length check (
encoded_labels), but the public-DNS-name return path rebuilds from the judged host only under the existing triggers —percent_decoded,has_literal_separator(#660),host_was_mapped(scoped to folds landing on ASCII, #665),has_ignored_codepoint(#652),has_extraction_fold(#667). A plain non-ASCII IDN host hits none of them and the function returnsrawverbatim. The Worker side ispkgs/lab-ops-hub/src/lib.ts:145(u.hostname, WHATWG-canonical punycode).Route: scoped fix enqueued as queue task 4848 (small_fix, dedupe
repo-health:v1:fix:developing-today/code:issue:659): rebuild the returned URL from the A-label form of the judged host when the judged host is non-ASCII, composing with the existing triggers and root-dot re-attachment; judgement policy, ASCII/IP/port/userinfo handling unchanged; tests pin café.com / bücher.example / 😀.com plus a Python↔TS IDN differential per this issue's acceptance criteria.58 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up (resume)
queue/4848-659-idn-alabel— this task's own canonical PR from its first attempt — and neither issue lab-ops-hub: container validator returns the raw U-label host for IDN targets — Worker returns the punycode A-label form it judged #659 nor PR fix(lab-ops-hub): return IDN hosts in A-label form from container validator (#659) #3781 carriesheld,user/owned,do-not-merge, ordo-not-auto-edit. Resuming the PR per the resume-don't-restart rule.fix(lab-ops-hub): apply compatibility mappings in A-label fallback (#659
Merge pull request #3781 from developing-today/queue/4848-659-idn-alabel
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done