Repository navigation
id(journals): a crash-torn final line that splits a multibyte UTF-8 character bricks the journal open — petnames, groups, outbox, artifacts and directory all read_to_string before their torn-tail handling runs #4369
Description
Problem
Every JSONL journal loader in pkgs/id that promises crash-torn-tail tolerance reads the whole file with std::fs::read_to_string before its torn-tail logic runs. read_to_string fails on any invalid UTF-8 anywhere in the file, so a crash that tears an append in the middle of a multibyte character (a name containing é, an emoji, CJK text — all legal in these journals) bricks the open entirely: the loader returns the UTF-8 error instead of dropping and truncating the torn final line as its own doc promises. The complete, valid records before the tail are unreachable.
Affected loaders (all verified at the lines below on current main):
crates/id-core/src/petname.rs:64—Petnames::open: doc (lines 54–58) promises a torn final line "is dropped and truncated away"; theread_to_stringat :64 errors first.src/groups/log.rs:56—GroupLog::open: same promise (doc lines 44–50), sameread_to_stringfirst.src/envelope_outbox.rs:135—Outbox::open: torn-tail handling at :143–175 is unreachable for a UTF-8-split tail.src/artifact.rs:579—ArtifactLog::open: doc (lines 573–576) promises a crash-cut final line "is dropped and the file is truncated";read_to_stringat :579 errors first.src/directory.rs:494—Directory::replay: the id(directory): a torn final line in directory.jsonl bricks the directory — Directory::replay refuses the crash-torn tail that ArtifactLog (loaded in the same open) drops and truncates #1521 torn-tail repair computeskept_lenfrom the string it already failed to read.
The world journal in the same codebase does this correctly: FileJournal::open (src/world_store.rs:59) reads bytes with read_until(b'\n') and parses with serde_json::from_slice, and read_journal_entries (src/world_store.rs:466) splits the raw bytes on b'\n' — a UTF-8-split torn tail is dropped there exactly as documented.
Evidence (verified this run)
Laptop probe (worktree ~/git/fleet/probe-2026-11-02-laneA-utf8torn, at GitHub main 9f096d9; all five loader sources md5-matched between the sandbox and laptop clones): an integration test against the real id_core::petname::Petnames:
- Control: a valid journal plus an ASCII torn tail (no newline) reopens fine — the existing torn-tail path works for ASCII.
- Probe: a valid journal whose first record sets the petname
José, followed by a torn second record cut after the first byte (0xC3) of anéin the name, with no trailing newline —Petnames::openFAILS instead of dropping the tail and keepingJosé.
Probe output (cargo test -p id-core --test probe_utf8_torn -- --nocapture):
CONTROL ascii torn tail: OK (dropped, Ali kept)
test control_ascii_torn_tail_is_dropped ... ok
PROBE utf8-split torn tail: open FAILED: stream did not contain valid UTF-8
test probe_utf8_split_torn_tail ... FAILED (BUG REPRODUCED: utf8-split torn tail bricks Petnames::open)
The other four loaders share the identical construction (read_to_string returning the same std UTF-8 error before any line logic), so the same torn file bricks each of them; the probe exercises the petname loader against the real crate code as the representative case.
Earlier torn-tail fixes — #1499/#2948 (petnames), #1521 (directory), #1522 (groups), #1620/#2150 (outbox), #2128 (artifacts) — each tested torn tails made of ASCII or complete characters, so this sub-case survived all of them.
Why it matters
These journals are the durable state of a world (directory, groups, outbox, artifacts, petnames). One crash at the wrong byte turns a recoverable, documented torn tail into a world that will not open at all, with an error ("stream did not contain valid UTF-8") that does not name the torn tail — the exact failure class the torn-tail fixes were written to eliminate, gated on whether the torn record happened to contain non-ASCII text.
Acceptance criteria
- Each affected loader reads bytes (or otherwise tolerates invalid UTF-8 confined to the unterminated final line), drops and truncates a torn final line even when it splits a multibyte character, and reopens with all complete records intact.
- A newline-terminated line containing invalid UTF-8 / unparseable content in any earlier position still fails the open as corruption (the torn-tail tolerance must not widen).
- Regression tests for each fixed loader: valid record with a non-ASCII name + a final line torn mid-character reopens with the valid records and the file truncated to the last complete line; the next append after reopen replays cleanly.
- Existing torn-tail and journal tests for petnames, groups, outbox, artifacts and directory still pass.
Activity
dezren39 commentedon Nov 2, 2026
🤖 Agent note · filed
dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
131 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · parked
opencode. Do not touch the PR, branch, or worktree.71134b8042fb6418370af6968fd519d26d5938d7, author dezren39, branchqueue/6242-4369-utf8-torn-journal, checks pass (basic line-limit pass, CodeRabbit skipping). Same head and ownership as the prior park.queue/6242-4369-utf8-torn-journaland worktree/home/user/git/fleet/6242untouched.dezren39 commentedon Nov 4, 2026
agent-note:v1
task: 6242
lane: code-y
stage: handed_off
execution_lane: opencode
Issue #4369 verified OPEN, no hard stops; parked labels resynced to queue/claimed + status/in-progress. Task 6242 re-claimed by lane code-y and handed off to the opencode claimant. RESUME, do not restart: PR #5233 is OPEN on branch queue/6242-4369-utf8-torn-journal — disposition its checks and Codex findings, merge when verified, then complete task 6242.
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · bound
queue/6242-…). The queue task is currently QUEUED and unclaimed — the original claim evaporated after the PR was opened — so this issue's claimed/assigned labels are being reset to the truth (queue/queued+status/needs-triage) in the same pass. The lane that claims task 6242 must BIND to PR fix(id): recover UTF-8-split torn journal tails #5233: checkpoint stagepr_openedwith the PR as artifact, park as its shepherd (review CI, merge when green), and NOT re-implement the work. (The monitor cannot claim a specific task through the fleet CLI —claimtakes no task selector andbindrequires a live claim — so the park itself is the claiming lane's first action.)pr-bound-shepherdat its next claim.