Repository navigation
Historian summaries: facts from after the range, one-character hash typos, and inline ordinal labels off by 4 #774
Description
Magic Context: 0.49.20 (summaries written over ~0.4x–0.49.x), OpenCode 2, historian on Claude Haiku.
We audited the historian's summaries in one long technical session (2,273 compartments): automatic checks of every hash, number and backticked token against the raw messages in each compartment's range, plus a hand read of ~18 compartments. Accuracy is good overall (no invented claims found; proposed-vs-done handled correctly in the hand-read sample). Three systematic issues:
- Facts from after the range leak into a compartment. 17 of 1,690 commit hashes (1.0%) first appear 15–64 messages after the compartment's end, e.g. "committed the fix as
<hash>" in a compartment whose range only contains the review and acceptance; the commit happens later. The claim is true, but attached to the wrong period, so it reads as done earlier than it was. A second session showed 14 such cases. This suggests the historian sees context past the cut; asking it to stay within the range (or mark later facts as later) would fix it. - One-character hash typos. 3 of 1,690 hashes (0.18%) are a real hash with one character changed (e.g.
d3714a6a9bford3714d6a9b) — non-existent commits inside otherwise-correct "committed X" statements. One number was also off (128.76 s for 128.68 s). A post-check that every hash/number in a summary appears in the raw range (with a 1-edit-distance correction) would catch all of these. - Inline ordinal labels are off by 4. The few
[N]labels the historian writes into summary text that we could verify (n=4, low confidence) were all exactly 4 lower than the real ordinal. Computing labels in code, or not writing them into the text, avoids this.
Also noted: "U:" lines are 88% exact quotes; the rest are lightly cleaned (typos fixed) or bracketed paraphrases, which is fine but worth documenting. Compartment ranges themselves are contiguous with no gaps or overlaps, and boundary IDs resolve to the stored ordinals.
Activity
magic-alfonso commentedon Oct 23, 2026
Thanks for this audit. A check of every hash and number against each compartment's own range across 2,273 compartments is far more thorough than anything we've run on our side.
The three findings are clear, and the first is the most important: a fact can be true and still read as done too early.
We're in the middle of a larger change to how the summarizer is built and prompted, and that work is where these belong, so we're queuing them rather than starting now. We'll post here when we pick them up. If you can share one or two of the after-the-range examples with message ids, with any private content removed, that will make them easy to reproduce.
null-axiom commentedon Oct 23, 2026
One after-the-range example (OpenCode 2 session, ordinals per the compartment row):
start_message=32501,end_message=32522,start_message_id=msg_0773c2e4d001iEX3TC3q0tleN5,end_message_id=msg_0774cb287001rj0ah3XGbSa40T(2026-09-06 15:00:27 → 15:18:30 UTC).402da53644. In the range, the worker only reported and the lead reviewed and accepted it.msg_07751113b001vh69djiy3GRrRA(assistant, 15:23:16 UTC), about five minutes after the range ends; the commit itself happens there.null-axiom commentedon Oct 24, 2026
Causes found:
compartment-runner-incremental.ts:~662, 790-797) and splits it into several compartments; validation only requires each compartment inside the chunk (compartment-runner-validation.ts:350). The example: run 8340 covers the chunk containing compartments seq 1147–1150, and the hash first appears inside seq 1149, in the same chunk. Thecommits: …header suffix lifted from assistant text (read-session-formatting.ts:547-555) likely makes hashes easy to attach to the wrong topic. Suggested: drop hashes that don't appear in a compartment's own range, or tell the historian a hash belongs to the compartment containing its block.chunk_start_ordinal32483, first compartmentstart_message32487;unprocessed_from32550 vs last end + 1 = 32554); the first OpenCode 2 run (10208) has 0 offset. 955 of 1,133 runs show +4. The rebase moved coordinates but didn't rewrite the[N]labels in summary text (24 of 30 labels checked have a user message at N+4). Suggested: rewrite labels by the same delta during the rebase, or don't put raw ordinals in summary text. Pre-rebase run telemetry is in the old numbering too.