Design: a value cannot act on the terminal — one format-character (Cf) contract for pkgs/id plain output

Design (design-architect pass 31, 2026-11-01). Full design doc: files/designs/code-id-plain-output-rendering.md in the keep-the-four-repos-healthy goal workspace. Audited at origin/main 70c20625. No user/review: every origin carries explicit acceptance with a verified probe, and the judgment calls below (one shared character set; escape vs strip by surface kind) are the architect's recorded decisions.

Summary

pkgs/id's plain-text ruling (#1613, recorded in src/commands/text.rs's module doc) is a printer never lets a value break row structure — but every guard in the family tests char::is_control() (category Cc only), so Unicode format characters (category Cf) reach the terminal raw and act on the rendering: bidi overrides/isolates (U+202A–U+202E, U+2066–U+2069) visually reorder the fields after them, and zero-width characters (U+200B, U+FEFF, U+00AD) make look-alike rows with different bytes — the Trojan-Source class applied to live CLI output. Three surfaces, one class:

id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839 — plain_field (commands/text.rs:42-44): a gossip peer name carrying only U+202E returns byte-identical unquoted; quoted, the override is still raw inside the quotes, before the blob-count/timestamp columns it can falsify.
id(group/doc/claim): the shared key=value renderer passes bidi/format controls (Cf) raw — a claim value with U+202E reaches the terminal unescaped in both the unquoted and quoted paths and visually reorders the row's author=/event= fields #3111 — the shared key=value renderer text() (commands/group.rs:64-72; group/doc/claim via tokens()): Cf is raw in both paths — the trigger is is_whitespace() || is_control(), and the quoted path is a serde_json literal, which never escapes Cf. Claim values are other nodes' signed tag events (ingest checks length + NFC only; Cf is NFC-stable) sitting before the row's real author=/event= fields.
id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089 — id world list's plain branch (commands/world.rs:683) prints host-supplied world names verbatim (neither renderer used): a newline forges a fully-formed extra row, a tab forges columns. Names arrive from a host's wire reply with no validation; id(peers): a peer's announced name is printed raw — a newline in a gossip announcement forges extra TSV rows in id peers and the REPL #1613's ruling exists precisely for non-conforming hosts.

The character set already exists in-tree, stated once: petname.rs:237-262 is_format_char is the complete Unicode 16 Cf range set, with the doc ruling that Cf characters "are invisible or reorder the text around them" — petnames refuse them at ingest (#1674), but the definition is a private fn the renderers never saw. world/util.rs:69-84's sanitizers (sanitize_display_name/sanitize_chat) share the is_control-only blind spot one layer down.

The contract (recorded decisions)

A plain-text field is row-safe only if it contains no character that can act on the terminal: no structure character (Cc controls, U+2028/U+2029) and no format character (category Cf, the crate's one shared set). Every plain renderer in pkgs/id accounts for both classes; the disposition differs by surface kind — row renderers escape, stripping sanitizers strip, ingest validators refuse — and no surface re-derives the character set.

One set: the petname set, shared. is_format_char is promoted to a single pub(crate) definition all phases use. It includes the zero-width characters id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839 left to triage (U+200B–U+200D, U+FEFF, U+00AD, U+2060): id(group/doc/claim): the shared key=value renderer passes bidi/format controls (Cf) raw — a claim value with U+202E reaches the terminal unescaped in both the unquoted and quoted paths and visually reorders the row's author=/event= fields #3111's look-alike argument covers them, and a renderer-specific subset is the per-site re-derivation this design stops. Recorded cost: a ZWJ-emoji or ZWNJ-joined name renders quoted-with-escapes in row renderers — content preserved in full, JSON-parseable — and petnames already refuse these characters outright. Reversible in one place if fleet evidence shows real names harmed.
Escape, don't strip, at row renderers; the two renderers stay distinct. Their trigger asymmetries are deliberate (space quotes in key=value, never in TSV). Cf joins each trigger and is rendered \uXXXX inside quotes (plain_field's existing form). For group::text the quoted path must escape Cf itself — serde_json's to_string() alone never will; that escaping is Phase 2's real work, not the trigger widening. Ordinary values render byte-identically; --json unchanged everywhere.
The printer, not ingest. No new ingest validation for peer names, claim values or world names — id(peers): a peer's announced name is printed raw — a newline in a gossip announcement forges extra TSV rows in id peers and the REPL #1613's ruling and id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089's non-conforming-host model make rendering the only total gate. Existing ingest checks stay as they are; the closing audit dispositions them separately.

Alternatives rejected in the doc: fixing the three sites independently as filed (the pattern that produced the family — #3111 exists because #2839's fix wouldn't touch its code); one unified renderer (trigger semantics genuinely differ; the shared artefact is the set + contract); stripping at renderers (destroys the printed-in-full, JSON-recoverable property and makes different values print identically); refusing at ingest as the primary fix (cannot clean stored values or non-conforming hosts; refusing other nodes' signed claim events is a federation decision); a bidi-only subset (misses the look-alike half, creates a second drifting set).

Phases (adopted origins — no parallel issues)

Phase 1 = id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839 (plain_field honours Cf; creates the shared set + text.rs doc truth-up; lands first).
Phase 2 = id(group/doc/claim): the shared key=value renderer passes bidi/format controls (Cf) raw — a claim value with U+202E reaches the terminal unescaped in both the unquoted and quoted paths and visually reorders the row's author=/event= fields #3111 (group::text honours Cf, incl. escaping inside the quoted path; gated on Phase 1 for the shared set definition only).
Phase 3a = id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089 (world list plain branch routes the name through plain_field; marker comparison stays on the raw name; not gated — correct today for structure characters, Cf coverage arrives with Phase 1).
Phase 3b = closing inventory + contract statement (filed as a sub-issue, gated on Phases 1–2): every remaining is_control-only guard in pkgs/id/src dispositioned escape / strip / refuse / deliberately-local with reason — starting population includes the world/util.rs sanitizers (expected: strip-Cf for sanitize_display_name; sanitize_chat decided against its downstream rendering), the directory/world/ssh validator sites, and world_presentation/world_session payload validation (id(web): world presenter widens every U+FE0F cluster to 2 — a letter+VS16 cell (a️) is Rust-valid at columns 1 but renderWorldSurface refuses the whole screen #3265's width mechanism is different and not adopted); contract + the set's home and Unicode version written into the text.rs module doc; closing evidence here, including the sibling-audit evidence id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839's acceptance requires.

Open questions stay in the design doc (ingest-side Cf refusal as defence in depth — federation decision, evidence-gated; sanitize_chat's final disposition if ZWJ-stripping would corrupt legitimate chat text; shared-set placement; ANSI/OSC terminal sequences — no filed instance).

Standing-candidate note: passes 28–30 named this family the standing candidate for this pass (displaced once at pass 30 by #3352's cluster, under its stated terms). Same-day clusters assessed and not taken: the aggregator impossible-date family is #2576's already-stated strict-date rule applied per site; the new operator value-option filings #3449/#3415/#3419 are #3352's contract — provider-catalog-sync is already a named candidate in #3353's closing inventory.

CODE-1321

Design filed this pass (2026-11-01, audit at origin/main 70c20625). Phases linked as sub-issues (verified count=4): Phase 1 = #2839 (adopted), Phase 2 = #3111 (adopted, status/blocked on Phase 1 for the shared set), Phase 3a = #3089 (adopted, not gated), Phase 3b = #3456 (new closing audit, status/blocked on Phases 1–2, triage packet → pilot task 5376 queued). Register entry added in files/potential-projects.md (status: proposed). No user/review — origins carry explicit acceptance with verified probes; the set/sharing decisions are recorded in the design doc.

Agent note (program tending, 2026-11-01): child progress — sub-issue #3089 (id(world): world list plain output renders host-supplied names raw) CLOSED 2026-11-01T14:25:41Z (parent link REST-verified this run). This epic's last comment (2026-11-01T09:35Z) predates the closure, so the landing is recorded here. Progress note only — not a completion proposal.

Design record — task 5751 triage of #3455 (Cf contract for pkgs/id plain output)

Pickup note and triage outcome in one: task 5751 (queue) claimed by lane E, revalidated against fresh GitHub state and current origin/main 73423be04, and worked to the record below. The full design doc is files/designs/code-id-plain-output-rendering.md in the keep-the-four-repos-healthy goal workspace (design-architect pass 31); this record is the decision-ready summary the triage packet requires, plus the packet wiring.

The decision to be made

Adopt one crate-wide contract for pkgs/id plain-text output:

A plain-text field is row-safe only if it contains no character that can act on the terminal: no structure character (Cc controls, U+2028/U+2029) and no format character (category Cf, the crate's one shared set). Every plain renderer in pkgs/id accounts for both classes; the disposition differs by surface kind — row renderers escape, stripping sanitizers strip, ingest validators refuse — and no surface re-derives the character set.

Evidence at current main (verified this run, `73423be04`)

pkgs/id/src/commands/text.rs — breaks_row is still c.is_control() || matches!(c, '\u{2028}' | '\u{2029}') (Cc + separators only); the escape loop renders any breaks_row char as \uXXXX inside quotes, so widening the predicate fixes trigger and escape together. The id(peers): a peer's announced name is printed raw — a newline in a gossip announcement forges extra TSV rows in id peers and the REPL #1613 ruling ("a printer never lets a value break row structure") is in this file's module doc.
pkgs/id/src/commands/group.rs — text() triggers on is_whitespace() || is_control() and renders the quoted path via serde_json to_string(), which never escapes Cf: Cf is raw in both paths (Phase 2's real work is the escaping, not the trigger).
pkgs/id/src/petname.rs:244 — is_format_char is the complete Unicode 16 Cf range set and is already pub(crate); its only consumers today are petname.rs itself and invite.rs — no renderer uses it yet. (This is one step ahead of the design doc's audit at 70c20625, which recorded it as a private fn; the promotion happened, the renderer wiring did not.)
pkgs/id/src/world/util.rs — sanitize_display_name / sanitize_chat still filter is_control() only: the sanitizer blind spot is live and belongs to the Phase 3b inventory.
pkgs/id/src/commands/world.rs — the world-list plain branch already routes the name through plain_field (Phase 3a landed; id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089 is CLOSED).

Options considered

Adopt the contract (recommended). One shared Cf set (petname.rs's is_format_char), escape at row renderers, phases land independently. This is the design doc's proposal.
Fix the three origin sites independently, as filed. Rejected: it is the pattern that produced the family — id(group/doc/claim): the shared key=value renderer passes bidi/format controls (Cf) raw — a claim value with U+202E reaches the terminal unescaped in both the unquoted and quoted paths and visually reorders the row's author=/event= fields #3111 exists because id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839's fix would not touch its renderer; the sanitizers and remaining is_control sites would be the next filings.
One unified renderer for all plain output. Rejected: TSV and key=value trigger semantics genuinely differ (the space asymmetry is deliberate and test-pinned in both modules); the shared artefact is the set + contract, not rendering code.
Strip Cf at the renderers. Rejected: destroys the printed-in-full, JSON-recoverable property and makes two different stored values print identically — the look-alike failure, inverted.
Refuse Cf at ingest as the primary fix. Rejected as primary: cannot clean stored values or non-conforming hosts, and claim values are other nodes' signed events (federation decision). Kept as an open question below.

Recommendation

Adopt option 1, with the design doc's three recorded decisions: (a) the petname set, shared, including the zero-width characters (U+200B–U+200D, U+FEFF, U+00AD, U+2060) — recorded cost: ZWJ-emoji / ZWNJ-joined names render quoted-with-escapes in row renderers, content preserved in full and JSON-parseable, reversible in one place; (b) escape, don't strip, at row renderers, renderers stay distinct; (c) the printer, not ingest — no new ingest validation. --json unchanged everywhere.

Invariants the implementation must hold

Ordinary values (spaces, CJK, emoji, accented letters, existing test corpora) render byte-identically; only Cf-carrying values change, and only by being quoted with \uXXXX escapes.
Phase 2 must not alter serde_json's existing escapes for Cc / " / \ — the group.rs corpus (test_space_free_values_render_byte_identically, test_json_output_is_unchanged, the id(claim/doc/group): plain-text output renders names/values raw — a newline in a tag name or value forges extra rows and duplicate author=/event= fields #1611 newline tests) is the regression net.
No surface re-derives the Cf set: one pub(crate) definition, Unicode 16 ranges, home and version named in the text.rs module doc by Phase 3b.
The default-marker comparison in world list stays on the raw name (already true post-id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089).

Phased packets — already enqueued (verified in the queue this run)

Phase	Issue	Queue task	Dedupe key	State
1 — `plain_field` honours Cf; shared set + `text.rs` doc truth-up	#2839	task 5439	`repo-health:v1:fix:developing-today/code:issue:2839`	queued
2 — `group::text` honours Cf incl. quoted-path escaping	#3111	task 5635	`repo-health:v1:fix:developing-today/code:issue:3111`	parked (gated on Phase 1 for the shared set)
3a — world list routes name through `plain_field`	#3089	—	—	done (issue closed 2026-11-01)
3b — closing `is_control` inventory + contract statement	#3456	task 5380	`repo-health:v1:fix:developing-today/code:issue:3456`	parked (gated on Phases 1–2)

No new packets were needed: the phase wiring the design called for already exists, so this triage enqueues nothing and duplicates nothing.

Questions for Drew

None blocking. One genuinely Drew-owned question exists but is explicitly non-blocking and evidence-gated, stated here so it is never a silent stall: should claim/peer ingest also refuse Cf (petname-style) as defence in depth? Options: (A) no — the render contract is total without it (recommended; revisit only on evidence of a non-terminal consumer being spoofed); (B) yes — refuse at ingest too, accepting that refusing other nodes' signed claim events is a federation-facing behaviour change. Recommendation: A. The remaining design-doc open questions (sanitize_chat's final disposition, shared-set placement) are implementation-phase judgments for the Phase 3b worker, not Drew decisions.

Agent note (task 5751, lane E): triage DONE. Design record posted above (#3455 (comment)). Phase packets verified already enqueued — task 5439 (#2839, queued), task 5635 (#3111, parked on Phase 1), task 5380 (#3456, parked on Phases 1–2); #3089 closed. No blocking questions for Drew. Claim labels removed; the epic stays open until its phases land.

Decision-ready record: design note; PR #3861.

Recommend one Unicode 16 Cf predicate in an unconditional crate-private module; current petname is world-gated, while the plain renderers are unconditional.
Keep renderer contracts distinct: plain_field uses its escaped field form; group::text() must also escape Cf explicitly in its quoted path.
World-list structure handling is already landed through plain_field, preserves raw-name marker comparison, and id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089 is closed; Phase 1 adds Cf coverage there.
Phase 1 (id(peers): plain_field passes bidi/format controls raw — a gossip peer name with U+202E reaches the terminal unescaped and visually reorders the row's columns #2839): relocate/reuse the shared predicate, wire plain_field, update the contract, and verify default/world-only/world-disabled builds.
Phase 2 (id(group/doc/claim): the shared key=value renderer passes bidi/format controls (Cf) raw — a claim value with U+202E reaches the terminal unescaped in both the unquoted and quoted paths and visually reorders the row's author=/event= fields #3111): update both group::text() paths; depends on Phase 1's shared boundary.
Phase 3a (id(world): world list plain output renders host-supplied names raw — a newline in a world name forges extra rows, and a tab forges columns #3089) is complete; Phase 3b (Phase 3b (audit) of #3455 — closing is_control inventory + plain-output contract statement #3456) follows Phases 1–2 to refresh and disposition the remaining guard inventory, update the module contract, and post closing evidence.
Acceptance preserves ordinary output and --json, keeps stored/raw values, and adds no peer/host/claim ingest validation as the primary fix.
Drew questions: none. The remaining sanitize_chat and ingest-defense points have non-blocking defaults recorded in the note.

Merge pull request #3861 from developing-today/design/id-3455

Program tending — sub-issue progress (2026-11-02)

Closed since this thread's last activity: #2839 (closed 2026-11-02T11:25Z).

Recorded for the parent rollup; this is a progress note, not a completion proposal — the parent stays open on its remaining children.

Phase 3b closing evidence (from #3456, queue task 5380)

Gates verified closed this run before starting: #2839 and #3111 both CLOSED; #3089 (Phase 3a) already closed. Full inventory table is posted on #3456 (#3456 (comment)). Summary for the design record:

Contract statement written into the pkgs/id/src/commands/text.rs module doc (PR Phase 3b (#3456): plain-output contract statement + terminal_text strips Cf #4458, open, not merged): the two classes (structure Cc + U+2028/U+2029; format Cf), the disposition-by-surface-kind rule (row renderers escape / stripping sanitizers strip / ingest validators refuse), the shared Cf home (crate::is_format_char, id-core/src/unicode.rs) + Unicode version 16.
One site fixed under the design on this branch: world_ssh::terminal_text now strips the shared Cf set in addition to Cc (with regression test). All other sites were either already conformant (escape/strip/refuse) or deliberately-local with a recorded reason.
One named follow-up filed, none dropped: Phase 3b follow-up (#3456): web/identity sanitize_name strips Cc + bidi subset only, zero-width Cf passes to collab display #4457 (web/identity sanitize_name partial Cf strip needs a product decision on ZWJ names).
id(web): world presenter widens every U+FE0F cluster to 2 — a letter+VS16 cell (a️) is Rust-valid at columns 1 but renderWorldSurface refuses the whole screen #3265 interaction recorded: none — U+FE0F is Mn, not Cf; width mechanism untouched per out-of-scope.

Sibling-audit evidence for the `world/util.rs` surfaces (#2839 acceptance)

#2839 requires the sibling surfaces (world/util.rs sanitizers) audited and either fixed in the same pass or explicitly filed. Phase 3b record:

sanitize_display_name (world/util.rs:69-76): already strips the full shared Cf set — the filter is !ch.is_control() && !crate::petname::is_format_char(*ch), and crate::petname::is_format_char is verified to be the same function as the shared predicate (id-core/petname.rs:16 re-exports crate::is_format_char = unicode.rs Cf set). No per-site re-derivation. Fixed-point property relied on by the id(world): journal restore never sanitizes Issued display names — empty, control-bearing, and over-length names restore verbatim, though mint refuses or strips all three #2960 restore check in world/core.rs is preserved. No change needed; no follow-up.
sanitize_chat (world/util.rs:95-103): already strips Cc (keeping LF/TAB) + all Cf except U+200C/U+200D, so legitimate joined-emoji text survives while bidi/zero-width actors are removed. This settles the design doc's recorded fallback question on the strict side (full strip minus joiners rather than bidi-subset), with the corruption rationale documented in the table. Downstream rendering decision: stored text is re-validated by ChatMessage::check and terminal display passes through terminal_text, now full-Cf. No change needed; no follow-up.

Remaining Phase 3b acceptance (checkbox 5) is this comment. PR #4458 carries the doc + fix for review; task 5380 parks on the PR.

Program tending — completion proposal (2026-11-03, via #1352)

All 4 sub-issues of this epic are now closed: #2839, #3111, #3089, #3456 (verified this run via the REST sub_issues read, cross-checked in a separate call). The Phase 3b closing evidence is already recorded on this thread (2026-11-02).

Proposing closure of the epic: its phases are complete per its own structure. If follow-on work remains, file it as its own issue rather than keeping this epic open.

agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/none). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03

Navigation Menu

Repository navigation

Design: a value cannot act on the terminal — one format-character (Cf) contract for pkgs/id plain output #3455

Summary

The contract (recorded decisions)

Phases (adopted origins — no parallel issues)

2 remaining items

Design record — task 5751 triage of #3455 (Cf contract for pkgs/id plain output)

The decision to be made

Evidence at current main (verified this run, `73423be04`)

Options considered

Recommendation

Invariants the implementation must hold

Phased packets — already enqueued (verified in the queue this run)

Questions for Drew

Program tending — sub-issue progress (2026-11-02)

Phase 3b closing evidence (from #3456, queue task 5380)

Sibling-audit evidence for the `world/util.rs` surfaces (#2839 acceptance)

Program tending — completion proposal (2026-11-03, via #1352)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Design: a value cannot act on the terminal — one format-character (Cf) contract for pkgs/id plain output #3455

Description

Summary

The contract (recorded decisions)

Phases (adopted origins — no parallel issues)

Activity

developing-today commented on Nov 1, 2026

dezren39 commented on Nov 1, 2026

dezren39 commented on Nov 1, 2026

2 remaining items

dezren39 commented on Nov 1, 2026

Design record — task 5751 triage of #3455 (Cf contract for pkgs/id plain output)

The decision to be made

Evidence at current main (verified this run, 73423be04)

Options considered

Recommendation

Invariants the implementation must hold

Phased packets — already enqueued (verified in the queue this run)

Questions for Drew

dezren39 commented on Nov 1, 2026

dezren39 commented on Nov 1, 2026

dezren39 commented on Nov 2, 2026

Program tending — sub-issue progress (2026-11-02)

dezren39 commented on Nov 2, 2026

Phase 3b closing evidence (from #3456, queue task 5380)

Sibling-audit evidence for the world/util.rs surfaces (#2839 acceptance)

dezren39 commented on Nov 3, 2026

Program tending — completion proposal (2026-11-03, via #1352)

developing-today commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Evidence at current main (verified this run, `73423be04`)

Sibling-audit evidence for the `world/util.rs` surfaces (#2839 acceptance)