[Pi Agent] Conflicting served tool tag numbers permanently block a session, reported as storage busy (0.47.0) #650

Activity

1. magic-alfonso commented on Oct 10, 2026

   

   magic-alfonsoBot

   on Oct 10, 2026

   More actions

   Thanks for the careful report and the read-only store inspection; the two rows (tag 8 on the real assistant-entry owner, tag 154 on a pi-msg- fallback owner, same tool call) are exactly what we need.

   Two things are clear already:

   • The refusal is correct to stop the turn: serving the history with a tool call carrying two different tag numbers would show the model inconsistent bytes. But the message is wrong. It says the storage is busy and to resend, while the actual cause is the identity conflict, which a resend can't fix. That wording is a bug.
   • There should be no path where the same tool call gets a second tag while its first one exists, so the conflict itself is a bug too, whatever triggered it.

   We're reproducing it now on Pi 1.1.0, including whether session restore or codemode plays a part, and looking at how a conflict like this can be repaired safely without breaking tag numbers the model has already seen. Your local repair (removing the duplicate active tag and keeping the original dropped one) matches what we'd expect to be safe in your case, but we'll confirm that before recommending it in general.

   We'll follow up here with the cause and the fix.
2. bin115885 commented on Oct 10, 2026

   

   bin115885

   on Oct 10, 2026

   Author

   More actions

   Follow-up: recurrence with a regular message-tag conflict

   The same primary Pi Agent session became blocked again after the earlier tool-tag repair. This time, the logged failure was a message-tag identity conflict rather than a tool-tag conflict:

   [2026-10-10T08:21:44.138Z] DEGRADED PASS Conflicting served Pi message tag numbers; refusing identity adoption:
   LKG unavailable (lkg_invalidated_reshape); refusing unreduced 315-message input
   

   The preceding pass reported:

   collectMessageEntryIdsByRef: resolved=313/315 (fingerprint=313, branchEntries=349, messageEntries=341)
   — unmapped slots fall through to synthesized ids
   transform: usage=87.5% (178473 tokens, limit=204000) decision=execute force=true
   pi m[0] HARD fold decision: reason=drift executed=true bustsServedPrefix=true
   

   Observed duplicate mapping

   Two type=message rows shared the same nonempty entry_fingerprint, including the same timestamp, role (user), and content hash:

   tag_number	message_id shape	status	byte_size	token_count
   20	actual Pi entry ID followed by :p0	active	233	103
   440	pi-msg-0-<timestamp>-user:p0	active	233	103

   Both source_contents rows were present and their content strings were exactly identical (221 characters). The original tag 20 had a pending drop operation; duplicate tag 440 had no pending operation.

   The installed adoptPiFallbackMessageTag implementation throws the message-tag error when multiple candidate rows are already in servedTagNumbers. This appears to be another instance of the fallback/real-identity reconciliation failure, now affecting an ordinary user message. A common root cause with the original tool-tag incident is not yet proven.

   Narrow local workaround

   In a single transaction, we removed only tag 440 and its identical source_contents copy. We retained tag 20 unchanged, its original content, and all pending operations. Assertions checked row identity/content equality before deletion and confirmed no remaining fallback-versus-real message mapping conflicts for matching fingerprints and part ordinals in this session.

   The transaction and store checks passed. Successful continuation of the actual Pi conversation after this second repair has not yet been verified. The Pi session JSONL and other sessions were not modified.

   This recurrence suggests the problem is not limited to tool results or codemode. As before, the trigger that first produces the duplicate mapping is unknown. Please let us know whether the message-tag case should be tracked separately.
3. magic-alfonso commented on Oct 10, 2026

   

   magic-alfonsoBot

   on Oct 10, 2026

   More actions

   Thanks, this is very useful. It's the same family as the tool-tag case, so we're tracking it here and fixing both together.

   What we've found so far, reproduced on real Pi hosts: when an extension earlier in Pi's context chain changes a message's text or the message count without changing its timestamp, Magic Context's owner matching could lose the real Pi entry id. It then gave the same message a second tag under a pi-msg- stand-in id. codemode takes the same path as ordinary tools and isn't required to trigger it, though we haven't confirmed what started it in your session.

   The fix in progress:

   • keeps the original identity when an earlier extension reshapes the context;
   • repairs an existing duplicate on its own, but only when the cached request proves which tag number the model actually saw. A pending drop like the one on your tag 20 is kept;
   • otherwise refuses with a message that names the identity conflict and says a resend won't fix it, instead of "storage is busy".

   It's under review now, and the message-tag case you just hit is being added to it. If it's easy on your side, it would help to know which other Pi extensions run in this session and their load order, especially any that edit, add or remove messages in the context hook.
4. Zzz210s commented on Oct 10, 2026

   

   Zzz210s

   on Oct 10, 2026

   More actions

   Another Pi host hitting the same family of refusals, with the load-order information @magic-alfonso asked for.

   Host: Pi 1.1.0, @cortexkit/pi-magic-context 0.47.0, Windows 11, Node 24.

   packages order in the Pi settings file (Magic Context is 5th):

   npm:pi-web-access
   npm:pi-cache-optimizer
   npm:pi-rtk-optimizer
   npm:@ff-labs/pi-fff
   npm:@cortexkit/pi-magic-context
   npm:@juicesharp/rpiv-todo@2.12.0
   npm:pi-scraper
   npm:@narumitw/pi-usage
   

   Two of them hook the context path ahead of Magic Context and mutate content:

   package	version	hooks	mutation
   pi-rtk-optimizer	0.9.0	tool_result, tool_execution_end, before_agent_start	rewrites/compacts tool output ("tool output compaction")
   pi-cache-optimizer	2.8.24	before_agent_start, tool_execution_end, before_provider_headers	rewrites the system prompt (session-overview churn strip, skill-index compression)

   Symptom here — same user-visible message, different stage than the report above:

   Magic Context could not safely prepare this turn; send your message again.
   stage=LKG/served publication elapsed=26994ms recovery=no managed result; refused
   
   DEGRADED PASS stage=writer admission elapsed=21186ms recovery=not attempted:
       LKG replay served 608 messages instead of raw 713
   turn refused stage=recovery elapsed=28841ms
   turn refused stage=pipeline elapsed=27112ms
   turn refused stage=LKG/served publication elapsed=25693ms
   SQLite boot busy timeout: backend=Node.js timeout=5000ms
   

   Store observations (read-only), in case they narrow it down:

   • No pi-msg- synthetic duplicates anywhere: zero message-fingerprint groups contain a synthetic id. The multi-row tool-owner groups that do exist look like ordinary multiple tool calls owned by one assistant message.
   • context.db is 701.6 MB with 40.7% free pages; the WAL had grown to 67.9 MB before I checkpointed it; integrity_check is ok; schema is v95 = supported_fence v95; every running process reports 0.47.0, so there is no mixed-version state.
   • 5 Pi sessions were running concurrently against that one database.

   I have not A/B-tested disabling either optimizer yet. Happy to do that and report back if it would help pin the trigger.
5. magic-alfonso commented on Oct 10, 2026

   

   magic-alfonsoBot

   on Oct 10, 2026

   More actions

   Thanks for the details, that helps.

   Your log shows a different failure from the one this issue reports, so it's best tracked on its own. There's no identity conflict and no pi-msg- duplicate. Your turns were refused because:

   • the database write lock stayed busy for about 21 seconds (stage=writer admission elapsed=21186ms), while 5 Pi sessions shared one context.db;
   • the last good request saved for that session was older than the current history (608 messages against 713). Replaying it would have dropped the newest 105 messages, so Magic Context refused rather than send that or the unmanaged history.

   The refusal is the intended safe outcome there, but a 21-second lock wait under 5 sessions shouldn't happen, and "send your message again" isn't helpful advice when it does. Could you open a separate issue with:

   1. the log lines from about a minute before the refusal, especially any that mention the historian, the dreamer, a migration, writer, lock or busy, from all five sessions if you can (each Pi process writes its own log);
   2. roughly what the five sessions were doing at that moment, for example whether one was compacting or running a long tool loop.

   The optimizer A/B test would still help this issue, #650. If disabling pi-rtk-optimizer or pi-cache-optimizer makes the duplicate-tag refusals stop, that would confirm the trigger we reproduced: an extension that rewrites messages in the context before Magic Context sees them.
6. Zzz210s commented on Oct 10, 2026

   

   Zzz210s

   on Oct 10, 2026

   More actions

   Follow-up: a reproducible instance of an extension rewriting tool output on this host.

   pi-rtk-optimizer 0.9.0 runs with "mode": "rewrite" and outputCompaction.enabled = true here. In this session, a shell tool call

   head -6 shared/skills/writing-plans/SKILL.md
   

   returns the file's frontmatter, then two lines taken from the middle of the same file, then a [201 more lines] marker — not the six lines that were asked for. The identical request written as head -n 6 <file> | cat, or as awk 'NR<=6' <file>, returns the correct six lines. So the plain head -N form is being rewritten and its output reassembled from elsewhere in the file.

   Reproduced three times in a row (two different files, same result). I have not yet A/B-tested whether disabling that extension clears the Magic Context refusals — happy to run it and report.
7. magic-alfonso commented on Oct 10, 2026

   

   magic-alfonsoBot

   on Oct 10, 2026

   More actions

   Thanks, that's a clear reproduction. Yes, please run the A/B when you can: the same kind of session with pi-rtk-optimizer disabled, and separately with only pi-cache-optimizer disabled, and tell us whether the refusals stop. An extension that rewrites tool output, or message content, before Magic Context sees it can make the same tool call look different from one turn to the next, which is the trigger we reproduced for the duplicate-tag refusals. The fix we're preparing makes Magic Context keep the original identity when that happens, instead of refusing the turn.
8. DGVY commented on Oct 10, 2026

   

   DGVY

   on Oct 10, 2026

   More actions

   Additional case: Oh My Pi host, no third-party context extension, 4 refusals across 2 concurrent sessions

   Thanks for the root-cause explanation. This adds an instance on a different host, plus a reverse-direction observation that may narrow the trigger.

   Environment

   Host	Oh My Pi (omp) 18.8.7, not Pi Agent 1.1.0
   Magic Context	@cortexkit/pi-magic-context 0.47.0 (npm plugin, installed under ~/.omp/plugins/)
   OMP extensions	pi-rewind 0.5.0 and @cortexkit/pi-magic-context 0.47.0 — those two only
   OMP extensions setting	[] (empty, no third-party extensions)
   OS	Linux 7.2.9-arch1-1, x64
   context.db	schema v95 = supported_fence v95; no mixed-version processes

   Symptom

   Four refusals in the log, across two concurrent sessions, all the same message-tag conflict:

   [13:47:51.520] [01a1260e-…] DEGRADED PASS Conflicting served Pi message tag numbers;
                  refusing identity adoption: LKG unavailable (lkg_invalidated_reshape);
                  refusing unreduced 30-message input
   [13:47:51.520] [magic-context][pi] turn refused Magic Context storage is busy; send your message again
   PiStorageBusyError: Magic Context storage is busy; send your message again
   
   [13:49:21.889] [01a1260e-…] … refusing unreduced 35-message input   (same session, 2nd)
   [13:52:52.344] [01a12613-…] … refusing unreduced 73-message input   (**other session**, 3rd)
   [13:53:09.941] [01a1260e-…] … refusing unreduced 40-message input   (same session, 4th)
   

   Two differences from the reports already on this issue:

   1. The LKG reason is lkg_invalidated_reshape, not lkg_miss. That matches the conclusion stated here — "an extension earlier in Pi's context chain … could lose the real Pi entry id".
   2. Cross-session: session A's second refusal (13:49:21) and session B's first (13:52:52) are three minutes apart, so this is not one session getting into a stuck state.

   Earlier passes already reported lost stable entry ids

   Several passes before the refusal, this session already logged unmapped slots at a stable ~7–8%:

   collectMessageEntryIdsByRef: resolved=32/37 (fingerprint=32, branchEntries=66, messageEntries=35)
   — unmapped slots fall through to synthesized ids
   collectMessageEntryIdsByRef: resolved=68/70 (fingerprint=68, branchEntries=115, messageEntries=69)
   collectMessageEntryIdsByRef: resolved=72/75 (fingerprint=72, branchEntries=123, messageEntries=74)
   

   Consistent with the stated root cause: some messages never get a real entry id, the entry-id set drifts between passes, and the LKG prefix check then fails (lkg_invalidated_reshape).

   Reverse evidence: no third-party context-rewriting extension on this host

   @Zzz210s's report involves pi-rtk-optimizer / pi-cache-optimizer, which rewrite messages ahead of the context hook. Neither is installed here. OMP's extensions setting is empty, and the only other plugin is pi-rewind (checkpoint/rewind). So:

   This family of refusals occurs with no third-party message-rewriting extension present.

   If the trigger really is "an earlier extension on the context chain rewrites message text or count", then on this host the source is either OMP itself (how it builds event.messages, or how it projects branch/compaction entries) or the pi-rewind checkpoint path. That may help scope the reproduction.

   Store observations (read-only)

   pi-served-tag-numbers/<session>.jsonl is append-only, and getPiServedTagNumbers returns the union of every recorded pass, so the set only grows:

   Session	Records	Union of numbers	Max
   01a1260e… (this session)	10	44	48
   01a12613… (concurrent)	31	87	91

   :pN parts of one logical message each hold their own number, and all of them are already in the served set, for example:

   tag 41  pi-msg-31-1791640116882-user:p0   active
   tag 42  pi-msg-31-1791640116882-user:p1   active
   tag 43  pi-msg-31-1791640116882-user:p2   active
   

   Among type=message rows, the share still in pi-msg-* fallback form: 16/27 in this session, 13/42 in the concurrent one — fallback → real adoption has been incomplete for a long time.

   Questions

   1. Does "it reproduces with no third-party extension" fit the trigger you reproduced? If not, should the hosts be tracked separately?
   2. Does the fix's second part ("repair an existing duplicate on its own, but only when the cached request proves which tag number the model actually saw") still help a session that has accumulated many served numbers? The historical union never shrinks, so would servedRows.length > 1 still hold there?
   3. I can supply the full plugin list from ~/.omp/plugins/package.json and the OMP version, plus more log lines from these sessions, if useful.

   (Session ids truncated; no full JSONL or database attached.)
9. magic-alfonso commented on Oct 10, 2026

   

   magic-alfonsoBot

   on Oct 10, 2026

   More actions

   Thanks, this is very useful. It fits, and the hosts don't need to be tracked separately.

   The trigger isn't third-party extensions as such. It's any message Magic Context can't match back to a saved session entry. Those messages get a temporary pi-msg-* id, and when the real id turns up on a later pass, the same message can end up with a second tag. Your resolved=32/37 ... unmapped slots fall through to synthesized ids lines are exactly that, and your table (16 of 27 message rows still in pi-msg-* form) shows how often it happened in your session. On your host the unmatched messages can come from Oh My Pi itself, for example messages it builds for the request that aren't saved entries, or from pi-rewind. An extension that rewrites messages is just one more way to get there.

   Two changes are coming, and both will be in the next release:

   1. Fewer unmatched messages. Magic Context now matches messages to saved entries by position, after checking each one's role, timestamp and ids, instead of hashing content. It falls back to the old matching on any mismatch, so ordinary passes leave far fewer slots unmapped. This is on master.
   2. Sessions with existing duplicates recover. To your second question: the repair doesn't need the served set to hold a single number. When the cached request shows which number the model last saw, that number is kept and nothing the model sees changes. When there's no such evidence, as in your lkg_invalidated_reshape case, Magic Context keeps the newest number for that message, rebuilds the request once, and records why. It never repairs the same message twice. A conflict it can't resolve gets an accurate "tag identity conflict" message instead of "storage is busy". This is in final review.

   Yes, please send the full list from ~/.omp/plugins/package.json and some more log lines. The most useful are the collectMessageEntryIdsByRef lines from the passes just before a refusal, plus anything logged when you used pi-rewind in those sessions, so we can confirm where the unmatched messages come from.
10. magic-alfonso commented on Oct 10, 2026

    

    magic-alfonsoBot

    on Oct 10, 2026

    More actions

    A fix for this is now on master and will ship in the next release.

    When one Pi message or tool call ends up with two tag numbers, Magic Context now handles it without getting stuck:

    • The number the model saw can be proven from the last request it sent: it keeps that number, and nothing the model has already seen changes.
    • It can't be proven: it rebuilds the cached prompt once, records it as a tag-identity repair (shown in /ctx-status and the logs), and never repairs the same message twice.
    • The same conflict comes back after that repair: it serves the turn with both numbers, as 0.47.0 did, and logs it. It no longer refuses the turn.

    The refusal text now says what happened instead of "storage busy". Messages that Pi or another extension adds without a saved session entry also keep their tag numbers when other messages move around them.

    We tested it with both setups described here: on Pi 1.1.0 with an extension that rewrites tool output, and on Oh My Pi 18.8.7 with pi-rewind installed and about 8% of messages unmatched. Every turn was served in both.
11. willowite commented on Oct 11, 2026

    

    willowite

    on Oct 11, 2026

    More actions

    Another host for this issue, with a trigger that may help confirm the master fix: Senpi (an OMP/Pi fork, @code-yeongyu/senpi 2026.10.10-11, Bun 1.4.2, Windows 11), @cortexkit/pi-magic-context 0.47.0, Magic Context owns compaction. Five refusals in two primary sessions over nine hours, all DEGRADED PASS Conflicting served Pi message tag numbers; refusing identity adoption: LKG unavailable (lkg_invalidated_reshape | lkg_miss).

    Every refused pass took the positional lane. Of 193 Pi passes, the 184 that logged collectMessageEntryIdsByRef: resolved=… were all served; the 5 refusals are among the 9 that did not. For each refused pass, the session file has exactly as many emitted entries (message, custom_message, branch_summary) as the pass's messages= count (41 and 43), so alignedEntryIds?.length === event.messages.length held and positions were trusted.

    On Senpi they never line up: its convertToLlm turns custom messages into user turns, drops some custom types, folds environment-context into the next user message, and other extensions inject turns without entries. The counts can still coincide. Store evidence from one session:

    • tag 24 <router-hint custom entry id>:p0 holds the first assistant message (its responseId and timestamp), which tag 7 already holds under its own entry id; tool tags 25-41 are owned by the entry before their real owner. All written by one positional pass.
    • tags 1-6 (aligned real ids) and 43-48 (pi-msg-0 … pi-msg-4) carry the same five head messages with identical entry_fingerprint, all served. A fingerprint pass makes the pi-msg tag, a positional pass re-keys it under a wrong real id, the next fingerprint pass makes a new pi-msg tag, and the next positional pass throws.

    One more data point on the LKG side: the positional pass that wrote tags 24-42 was a forced materialization that also spawned the historian, and its LKG/served publication stage took 80.7 s. Its queued LKG capture and ledger line were discarded, but its numbers were already in pi-served-tag-numbers, so the next refusal had no LKG to replay (lkg_invalidated_reshape).

    Locally we ported the findPiAlignmentMismatch header check from master in front of the 0.47.0 positional lane, and made the two "Conflicting served" throws skip the adoption and keep both numbers. With the check, Senpi passes always take the fingerprint lane (custom messages arrive as user, not custom). Is a patch release with the #650 fix planned soon? Happy to run it on this host.
12. magic-alfonso commented on Oct 11, 2026

    

    magic-alfonsoBot

    on Oct 11, 2026

    Last edited by magic-alfonso

    More actions

    Thank you, this trace is very useful. It matches what the master fix changes:

    • Positions are trusted only when each message also matches its saved entry by role, timestamp and call ids (the header check you ported). On Senpi, where custom messages arrive as user turns and other turns have no entry, that check fails and the pass takes the fingerprint lane, as you saw.
    • A conflict between served tag numbers no longer refuses the turn over and over. If the last served number can be proven it is kept with no byte change; otherwise the session gets one recorded rebuild, and any later recurrence is served and logged instead of refused. A conflict refuses at most one turn.
    • Refusals for this case now say "tool-tag identity conflict" instead of "storage busy".

    Your observation about the forced pass that spent 80.7 s in LKG/served publication, and left its served numbers stored while its last-good capture was discarded, is a separate point. Pi passes are much cheaper on master, but I haven't confirmed that the served numbers and the last-good request are always saved together, so I'm checking that rather than assuming it.

    The fix is on master and ships in the next release. We'll post here when it's out, and a run on your Senpi host then would be a good confirmation.
13. DGVY commented on Oct 11, 2026

    

    DGVY

    on Oct 11, 2026

    More actions

    Requested data (OMP host)

    1. Plugin list. OMP reads plugins from ~/.omp/plugins/package.json + omp-plugins.lock.json:

    {
      "name": "omp-plugins",
      "private": true,
      "dependencies": {
        "@cortexkit/pi-magic-context": "0.47.0",
        "pi-rewind": "npm:pi-rewind"
      }
    }

    {
      "plugins": {
        "pi-rewind": { "version": "0.5.0", "enabledFeatures": null, "enabled": true },
        "@cortexkit/pi-magic-context": { "version": "0.47.0", "enabledFeatures": null, "enabled": true }
      },
      "settings": {}
    }

    OMP's own extensions setting is []. So the context chain ahead of Magic Context holds only pi-rewind 0.5.0 — matching the setup you tested.

    2. Version correction. When I filed the report the host was OMP 18.8.7; it has since auto-updated to 18.8.9 (~/.omp/natives/18.8.9/). The refusals below were all on 18.8.7.

    3. collectMessageEntryIdsByRef lines — caveat. The passes immediately before the four refusals are gone: the per-process log at /tmp/omp/magic-context/magic-context.log has rotated and now starts at 2026-10-11T16:08Z (~51 KB). What survives are same-shape lines from the same two sessions today, ~4.5% unmapped:

    [01a12613-…] collectMessageEntryIdsByRef: resolved=149/156 (fingerprint=149, branchEntries=254, messageEntries=158)
    [01a12613-…] collectMessageEntryIdsByRef: resolved=151/158 (fingerprint=151, branchEntries=258, messageEntries=161)
    [01a12613-…] collectMessageEntryIdsByRef: resolved=154/161 (fingerprint=154, branchEntries=263, messageEntries=164)
    

    For reference, the refusal-time lines I quoted in my first comment were resolved=32/37, resolved=68/70, resolved=72/75 — a similar 7–8% miss rate.

    4. pi-rewind logging — nothing to report. Searching the Magic Context log for rewind, checkpoint or restore returns no matches in either session, and I found no separate pi-rewind log directory under ~/.omp/. I used no rewind/checkpoint restore during those sessions, so I can't yet confirm whether rewind is the source of the unmatched messages on this host. If you know where pi-rewind writes diagnostics, I'll collect from there.

    5. One observation from the surviving session file. The concurrent session 01a12613-… (the 73-message refusal) still has its JSONL. Its entry types:

    entry type	count
    message	175
    custom	103
    title / session / thinking_level_change / service_tier_change / title_change	1 each

    Only message entries are emit-eligible here (custom is not custom_message), and the file has emitted-entry counts in the same range as the pass messages= counts (156/158/161). That is the shape where alignedEntryIds?.length === event.messages.length can hold while positions are not trustworthy — the case @willowite describes on Senpi, presumably via OMP's own request construction rather than a rewriting extension.

    I can attach entry-id/position pairs from that JSONL for the refused pass if that would help confirm it, and I'm happy to re-run on 18.8.9 once the release ships.

    (Session ids truncated; no full JSONL or database attached yet.)
14. magic-alfonso commented on Oct 11, 2026

    

    magic-alfonsoBot

    on Oct 11, 2026

    More actions

    Thank you, that's exactly what we needed, and you don't need to send the entry-id pairs.

    Your observation matches the other report: a pass where the counts happen to line up while the positions don't. On master, an equal count is no longer enough to trust positions. Each message must also match its saved entry by role, timestamp and call ids, and any mismatch falls back to content matching. With pi-rewind and no restores used, the 4.5–8% unmatched slots on your host are most likely messages Oh My Pi builds for the request that have no saved message entry. On master those are matched by content and don't break the session.

    We'll post here when the release with the fix is out. A run on 18.8.9 then would be a good confirmation.
15. magic-alfonso commented on Oct 12, 2026

    

    magic-alfonsoBot

    on Oct 12, 2026

    More actions

    Fixed in 0.48.0. A conflicting served tag number now resolves without permanently blocking the session: Magic Context keeps the number the model already saw where it can prove it, otherwise performs one recorded rebuild, and never refuses the same conflict more than one turn. The refusal no longer says 'storage busy' for this case. Pi also now saves the served tag numbers and the last good request together, so a slow pass can't leave one without the other. Thanks to everyone who traced this.
16. magic-alfonso

    closed this as completedon Oct 12, 2026