Skip to content

[Pi] PiStorageBusyError on SQLite lock contention triggers fail-closed ctx.abort(), blocking interactive turns #601

Closed

Description

@zhexulong

What happened?

In @cortexkit/pi-magic-context (v0.44.4) on Windows 11, when concurrent background maintenance (such as session_project_backfill, dreamer smart-note sweeps, or LKG pruning) holds a write transaction on a large context.db (~2.6GB), any interactive turn that encounters SQLITE_BUSY (database is locked) fails closed instead of failing open:

[magic-context][pi] turn refused Magic Context storage is busy; send your message again
PiStorageBusyError: Magic Context storage is busy; send your message again
    at assertPiRawFallbackFits (D:/PiData/agent/npm/node_modules/@cortexkit/pi-magic-context/dist/index.js:28967:11)
    ...

The user is shown:
Magic Context could not safely prepare this turn; send your message again.
and ctx.abort() is called, immediately cancelling the user's interactive turn.

Reproduction & Log Evidence

  1. Run Pi with @cortexkit/pi-magic-context on Windows 11 with an accumulated context.db over 1GB.
  2. Background operations start holding transactions:
    [magic-context] slow write transaction: site=session_project_backfill held=3208.9ms
    [magic-context] slow write transaction: site=embedding_stale_gc held=1064.3ms
    [dreamer] compiled smart-note sweep failed: database is locked
    
  3. A user message arrives during this lock window. In assertPiRawFallbackFits:
    if (!contextLimit || !Number.isFinite(contextLimit) || !observed || ...) {
      log("raw_fallback_refused completeness=partial");
      throw new PiStorageBusyError({ cause });
    }
  4. The error propagates to registerPiGuardedContext:
    pi.on("context", async (event, ctx) => {
      try {
        return await handler(event, ctx);
      } catch (error) {
        log("[magic-context][pi] turn refused", error);
        if (typeof ctx.abort !== "function") throw error;
        try {
          pi.appendEntry(ENTRY_TYPE, { message: RETRY_MESSAGE });
        } finally {
          ctx.abort();
        }
        return { messages: event.messages };
      }
    });
  5. ctx.abort() aborts the active Pi turn.

Expected Behavior

Unlike OpenCode where storage contention fails open (the pass continues or operations defer to the next natural cache bust), in Pi transient PiStorageBusyError should NOT abort the user's turn. It should either:

  • Degrade gracefully / fail open to passing raw messages through unmodified ({ messages: event.messages }) without calling ctx.abort().
  • Or retry with a brief exponential backoff before refusing.

Technical Analysis & Suggested Fix

  1. In packages/pi-plugin/src/pi-context-refusal.ts:
    Check the error type in registerPiGuardedContext. If error instanceof PiStorageBusyError (or isTransientPiStorageError(error)), do not call ctx.abort(). Instead, log a warning and return { messages: event.messages } so the user is never blocked from interacting with the agent.
  2. In packages/plugin/src/features/magic-context/storage-db.ts:
    Ensure background maintenance transactions (such as session_project_backfill) are batched into small slices (<100ms) rather than holding continuous multi-second locks on Windows.
  3. Database Maintenance:
    Provide a command or automated retention prune to prevent context.db from growing into multi-gigabyte files that amplify lock contention on Windows.

Environment

  • OS: Windows 11 (NTFS)
  • Package: @cortexkit/pi-magic-context@0.44.4
  • Harness: Pi 0.99.x
  • Client: Pi

Activity

  1. magic-alfonso commented on Oct 2, 2026

    @magic-alfonso

    Thanks for the detailed report and the logs.

    The refusal itself is deliberate, and we'll keep it. When storage is busy, Magic Context first tries to resend the last request it built for the session, then retries for the lock, and only then refuses the turn. Passing the unreduced conversation through instead would send everything Magic Context had removed, which can push a long session past the model's context window, and it costs a full prompt-cache rebuild on that turn and again on the next. A refusal costs one resend.

    The real problem in your log is that background maintenance holds the database for seconds at a time: session_project_backfill held=3208.9ms and embedding_stale_gc held=1064.3ms. A turn that arrives during one of those holds runs out of retries. Background work shouldn't hold the lock that long, so we're changing these jobs to commit in small batches. We're also checking why your turn reached the refusal instead of resending the last request. The fix is planned for the next release.

    Two things would help:

    • Does this happen mostly in the first minutes after starting Pi, or throughout a session? session_project_backfill is a one-time job after an upgrade, so if the refusals stop once it finishes, that narrows it down.
    • Do you still get refusals after the backfill has completed? If so, the slow write transaction lines from around those times would tell us which job is responsible.

    On the database size: the next release also prunes saved requests for sessions idle for more than a day. On our own store that removed about 700 MB.

  2. magic-alfonso commented on Oct 4, 2026

    @magic-alfonso

    Fixed in 0.45.0, now on npm. Background maintenance now holds the database for short stretches only: registering embeddings went from about 210 ms to under 2 ms on a 7 GB store, old saved requests are deleted in small pieces, and reading notes no longer takes the write lock. A Pi turn waits at most 250 ms for the database, then sends its last good request again or refuses the turn. As discussed, it never sends the unreduced history.

    To update, quit every OpenCode and Pi process first: the database migrates on first start, which can take up to a minute on a large database. OpenCode 1 and Pi pick up the new version on restart. On OpenCode 2 (2.0.22 or newer is now required), open /plugins, select Magic Context and press ctrl+u, or run opencode plugin update. If you still see this on 0.45.0, please reopen with a log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions