Any read transaction left open on statev2.sqlite freezes every server write once the WAL passes 32 MB

What happened

The desktop app froze ("T3Code is stuck"). For about 15 minutes threads and the sidebar did not load, and nothing the server writes went through. It recovered within a minute, without a restart, once an outside process that was reading statev2.sqlite exited.

Diagnosis

Since #21799 the checkpoint worker gates writer admission. gated() (packages/shared/src/nodeSqliteClient.ts:655) holds every write while blocked is true. The worker sets blocked when the WAL is at the 10,000-frame ceiling or the -wal file is larger than journal_size_limit (32 MB, apps/server/src/persistence/Sqlite.ts:12), and the checkpoint can't reach the end of the log.

A checkpoint can't move past the snapshot of an open read transaction. So once the WAL passes 32 MB, any connection that keeps a read open keeps blocked true, and every write waits for as long as that read lasts. runWithAdmission (line 794) has no deadline. The wait also happens while the statement holds the writer connection's lock (lines 1118 and 1026), so reads routed to the writer queue behind it too.

Holding writes doesn't help the checkpoint here: the reader blocks it, not the writes. So the gate turns WAL growth, which SQLite handles fine, into a full stall. A fix probably means not gating on blocked while a reader pins the WAL, or giving admission a short deadline after which writes go through and the WAL grows.

On this machine the reader was migrate-dev-db, run by an agent in a worktree, copying the live 17 GB statev2.sqlite in one read transaction. AGENTS.md says reading and copying the live database is fine, so agents do this routinely. A user script that snapshots a few tables every hour, holding a read for 60 to 171 s, is a second trigger on this machine. The server's own reader connection can pin the WAL the same way, because any open read does.

Steps to reproduce

Save this as packages/shared/repro-wal-gate.ts on main and run it from packages/shared:

// Repro: an outside read transaction stalls every T3 write once the WAL passes journal_size_limit.
// Save in packages/shared, then: node repro-wal-gate.ts worker   (or: noworker)
import * as NodeSqlite from "node:sqlite";
import { mkdtempSync, statSync } from "node:fs";
import { tmpdir } from "node:os";
import { randomBytes } from "node:crypto";
import * as Effect from "effect/Effect";
import * as Exit from "effect/Exit";
import * as SqlClient from "effect/sql/SqlClient";
import * as NodeSqliteClient from "./src/nodeSqliteClient.ts";

const checkpointWorker = process.argv[2] === "worker";
const filename = `${mkdtempSync(`${tmpdir()}/t3-wal-gate-`)}/statev2.sqlite`;
const walMb = () => (statSync(`${filename}-wal`, { throwIfNoEntry: false })?.size ?? 0) / 1048576;
const payload = randomBytes(64 * 1024);

const program = Effect.gen(function* () {
  const sql = yield* SqlClient.SqlClient;
  // Same pragmas as apps/server/src/persistence/Sqlite.ts.
  yield* sql`PRAGMA busy_timeout = 5000`;
  yield* sql`PRAGMA journal_mode = WAL`;
  yield* sql`PRAGMA synchronous = NORMAL`;
  yield* sql`PRAGMA checkpoint_fullfsync = ON`;
  yield* sql.unsafe(`PRAGMA journal_size_limit = ${32 * 1024 * 1024}`);
  yield* sql`CREATE TABLE t (id INTEGER PRIMARY KEY, data BLOB)`;
  yield* sql`INSERT INTO t (data) VALUES (${payload})`;

  // An outside reader, like `sqlite3 statev2.sqlite` or a backup script.
  const outside = new NodeSqlite.DatabaseSync(filename, { readOnly: true });
  outside.exec("BEGIN");
  outside.prepare("SELECT count(*) FROM t").get();
  console.log(`checkpointWorker=${checkpointWorker}: outside reader holds a read transaction`);

  for (let i = 1; i <= 1500; i++) {
    const started = performance.now();
    const exit = yield* Effect.exit(sql`INSERT INTO t (data) VALUES (${payload})`.pipe(Effect.timeout("5 seconds")));
    if (Exit.isFailure(exit)) {
      console.log(`write ${i} STALLED for 5s+ (WAL ${walMb().toFixed(1)} MB)`);
      outside.exec("COMMIT");
      const resumed = performance.now();
      yield* sql`INSERT INTO t (data) VALUES (${payload})`;
      console.log(`outside reader closed; next write took ${(performance.now() - resumed).toFixed(0)} ms`);
      return;
    }
    if (performance.now() - started > 500) console.log(`write ${i} slow: ${(performance.now() - started).toFixed(0)} ms`);
  }
  console.log(`all 1500 writes finished with the reader open (WAL ${walMb().toFixed(1)} MB)`);
  outside.exec("COMMIT");
});

await Effect.runPromise(
  program.pipe(
    Effect.provide(NodeSqliteClient.layer({ filename, readerWorker: true, checkpointWorker })),
  ),
);

$ node repro-wal-gate.ts worker
checkpointWorker=true: outside reader holds a read transaction
write 552 STALLED for 5s+ (WAL 39.3 MB)
outside reader closed; next write took 102 ms

$ node repro-wal-gate.ts noworker
checkpointWorker=false: outside reader holds a read transaction
all 1500 writes finished with the reader open (WAL 106.9 MB)

Expected: writes keep going while an outside reader is open, and the WAL grows until the reader leaves, as SQLite's WAL mode allows. Actual: once the WAL passes 32 MB, writes stop until the reader closes. The stall hit write 552 on both runs.

Version

Desktop 0.1.2-nightly.20261021.3499. The local server reports the same serverVersion. Line references are from main @ 5ffe045, which has no later change to nodeSqliteClient.ts.

Environment

macOS 26.6.2 on an Apple M4 MacBook Air with 16 GB. Desktop app with its bundled server (Electron 44.9.3, Node 24.22.0, SQLite 3.53.4). statev2.sqlite is 17 GB. About 14 Claude and Codex threads were running.

Evidence

sqlite spans on the writer connection from server.trace.ndjson, in local time (UTC+4). The outside copy started at 11:56 and was killed at about 12:11. Each duration is almost entirely db.lock_wait_ms, and all of them completed within a second of the kill.

start     durationMs  connection  db.lock_wait_ms  query
11:56:30  901121      writer      901120           SELECT * FROM scheduled_tasks WHERE enabled = 1 AND next_run_at IS NOT NULL ...
11:56:34  897756      writer      897755           SELECT MAX(sequence) AS sequence FROM orchestration_events INDEXED BY ...
11:56:48  883879      writer      883878           SELECT project_id AS "projectId", title, workspace_root AS "workspaceRoot", ...
11:57:28  844924      writer      844922           SELECT thread_id, transcript_imported_at FROM orchestration_v2_legacy_imports ...
12:01:09  624913      writer      624913           SELECT session_id AS "sessionId", subject AS "subject", scopes AS "scopes", ...

50 writer statements waited more than 10 minutes, the longest 901 s. From 11:57 to 12:10 the server started only 1 to 18 SQL statements a minute, against 1,000 to 2,800 a minute before. In the minute after the reader exited it ran 5,072, and after that every statement took 2 ms or less. At this event rate the trace keeps only about 20 minutes (10 MB files rotated every two minutes), so an earlier 104 s stall at 11:50 rotated out before I could classify it.

Related issues

fix(shared): checkpoint WAL without blocking the event loop #21799 added the admission gate.
Opening a long thread blocks the database writer for 7–10 s #21840 (open): snapshot reads that hold the writer stall writes for 7 to 10 s. Same symptom, different cause: this stall needs no T3 read at all, and it lasts as long as the outside reader does.
Local Windows desktop disconnects during 36–44 second server stalls with a 12.5 GB database #21539 (closed): 36 to 44 s server stalls on Windows with a 12.5 GB database. Similar symptom; I did not check whether a reader was open there.

Fix applied or workaround

I killed the outside reader. The server drained its backlog within a minute and needed no restart. The local stopgap is to turn the checkpoint worker off (checkpointWorker: null in the bundled server's SQLite layer), which falls back to SQLite's passive auto-checkpoint. The noworker run above shows that path doesn't stall.

Filed by

Claude Code (Claude Opus 5.5), following .github/triage/PLAYBOOK.md.

It froze again 40 minutes later with no outside process holding the database; only the server had statev2.sqlite open. This time the reader was the server's own shell snapshot. loadShellSnapshot ran a sql.transaction on the reader connection for 39 to 66 s while the machine was under load (load average 15) and the WAL was 33.7 MB. Writer statements waited up to 78 s for the writer lock until that read ended. It happened from 12:49 to 12:51 and again from 12:52 to 12:53 (UTC+4).

start     end       s     span                                 connection  lock wait s
12:52:29  12:53:27  58.2  sql.transaction (loadShellSnapshot)  reader      0.0
12:52:42  12:53:37  55.4  sql.transaction                      writer      55.4
12:52:48  12:53:38  50.4  sql.transaction                      writer      50.4
12:53:00  12:53:39  38.9  sql.transaction                      writer      38.9

So the gate needs no outside reader. Once the WAL is past 32 MB, any slow read on the server's own reader connection freezes writes for as long as that read lasts.

Another data point from a large Windows install, in case it helps size this.

Setup

T3 Code Nightly 0.1.2-nightly.20261021.3473, desktop on Windows 11.
16 logical CPUs, 64 GB RAM.
statev2.sqlite is 31.1 GB with about 2,790 threads. 959 are unarchived, after I archived 1,829 finished ones today.
10-20 agent threads run at once, with heavy MCP orchestration (about 17-28 tool calls a minute).

Stalls

From 20 Oct 22:31 to 22 Oct 13:30, event-loop-stalls.ndjson has 1,470 records. Median max delay is 4.6 s and p90 is 9.7 s; 22 records are over 20 s and the worst is 79.9 s. Utilisation median is 0.93.
Two full client resubscribes (the "Reconnecting" banner) at 13:23:51 and 13:27:31 lined up with 10.8 s and 8.5 s stalls.
Archiving the 1,829 threads didn't help. From 19:53 to 21:31 there were 67 records: median 7.2 s, p90 17.6 s, worst 27.0 s.

Where the time goes (trace, 19:53-21:30, UTC+5)

Writer-connection sqlite spans: 17,940.
- 97.3% of their total duration is db.lock_wait_ms.
- Lock wait: p99 5.7 s, max 90.8 s.
- Execution: p50 about 0 ms, p99 384 ms.
28 sql.transactions executed for 5 s or more, not counting lock wait:
- reader: ThreadPullRequestServiceV2.synchronize 7 times, up to 120.7 s, and ShellStream.loadShellSnapshotParts 3 times, 31-32 s, via loadShellSnapshot and subscribeShell;
- writer: Orchestrator.dispatchMessage (from sendToThread and dispatchContinuation) 42-53 s; Orchestrator.finalizeAppOwnedSubagentTask 29-69 s; providerTurnStart.start 31.5 s; OrchestratorMcpService.readThread/loadReadableThread 25.5 s.
The -wal file was 18.9-29.8 MB each time I sampled it. So I can't tell whether the 32 MB gate fired here, but the long reader transactions above are the kind that would pin it.

I can share span trees with timings only, no thread content, if useful.

Related: #23372 and #23378.

same here

Thanks for the detailed diagnosis and the repro script. They made our stall easy to place. I hit the same freeze last night, with a different outside reader: a read-only dbstat size scan of a 54 GB statev2.sqlite. All server writes stopped for 7 minutes, and they resumed in the same second the scan finished. I haven't tested #23496 against this. The trigger looks like the pinned-reader case it handles.

Trace evidence (traced by Claude Opus 5.5)

Environment: T3 Code Nightly desktop 0.1.3-nightly.20261027.3748 (Electron 44.9.3, embedded Node 24.22.0, SQLite 3.53.4), macOS 27.0.1, Apple M3 Pro (Mac15,7), 36 GB. statev2.sqlite is 54.1 GB (page_count 13,215,276 × 4096, freelist_count 0).

Timeline (UTC, 2026-10-27):

03:26:23: an agent starts sqlite3 -readonly "file:…/statev2.sqlite?mode=ro" "select name, sum(pgsize) … from dbstat group by name …" > t3-dbstat.txt (sqlite3 CLI 3.54.0). It is one statement, so one read transaction for the whole scan.
03:35: the desktop app restarts its backend (the 60 s readiness wait times out, but the server stays up).
03:37:13: writer waits begin.
03:44:22: t3-dbstat.txt gets its final write (file mtime), and every queued writer wait completes.

From server.trace.ndjson:

29 writer-connection spans waited longer than 60 s (81 to 428 s, almost all of it db.lock_wait_ms), and all 29 ended at 03:44:22. Examples: sql.transaction under orchestrationV2.EventSink.commitCommand for thread.visit (ws.rpc.orchestration.dispatchCommand took 428.9 s), thread.pull-request-link.sync, and UPDATE scheduled_task_webhook_pending … (db.lock_wait_ms 426927).
All 29 report the same db.lock_owner_at_start: fiber 62601, serverActivation.forkParkedFiber. That is an EffectWorker.runDaemon fiber outside any execute span, so its writer statement is most likely the untraced EffectOutbox.claimNext claim UPDATE, which waits in runWithAdmission while it holds the writer permit.
No statev2.sqlite writer statement completed between 03:37:13 and 03:44:22. Reader-connection queries kept completing (for example at 03:38:46 and 03:39:05), so the event loop was alive. The writes were held at admission.
The client showed "Some requests are slow". Eight orchestration.dispatchCommand requests started between 03:37:13 and 03:43:01 took longer than 15 s.

Adding an independent reproduction on current upstream main: 9e09926d97824179d97311d1a810c6b5bb60d2d1.

I applied only the regression tests from #23496 (head 2f83c052b0f64f6166d96073751a30f03c7a1994) to that main checkout, leaving its implementation unchanged:

vp test run packages/shared/src/nodeSqliteClient.test.ts \
  -t 'keeps admitting writes while an outside read transaction pins'

Both synthetic cases fail:

Scenario	Result on unchanged main
Reader prevents PASSIVE backfill (`readerAfter=0`)	Write 39 held by admission
Reader prevents TRUNCATE (`readerAfter=38`)	Write 39 held by admission

In both cases, the assertion reports a 41,517,272-byte WAL against a 33,554,432-byte journal_size_limit. Applying only the SQLite implementation patch from the same PR to that checkout makes the entire SQLite client suite pass: 20/20. Test runtime: Node 26.12.1 / SQLite 3.54.0 / Effect 4.0.5.

The tests use temporary synthetic databases and worker-message synchronization. No production database, credentials, conversation content, or production logs are included here.

This supports #23496's approach: reopen writer admission when a reader pins PASSIVE backfill or TRUNCATE, retain per-write ceiling coordination, and keep idle retries so the WAL can be reclaimed after the reader closes without requiring another write. Please consider merging that fix or an equivalent implementation with these regressions retained.

This confirms the admission-gate defect on the tested main revision; it does not establish that every event-loop stall or reader-queue delay has the same cause. A restart alone does not remove this reproducible source-level condition.

Note

🤖 Claude Opus 5.5 responding on behalf of Philip.

Another data point, from a headless server rather than the desktop app. It shows how long the stall can last when the outside reader is an orphaned process.

Environment

t3 serve as a systemd user service on Linux x64, version 0.1.6-nightly.20261029.3892, running since Oct 29 without a restart.
Around 20 coding agents (Codex and Claude) on one server, each started and woken through orchestration.dispatchCommand.
statev2.sqlite is 20.2 GB.

What happened (Nov 3, UTC)

At 01:29 a background script from one of our own agents opened a read-only connection to statev2.sqlite. Its parent exited, but the python3 child kept running and kept the read open.
From 01:33:57, sql.transaction ran for 1,551 s. orchestrationV2.EventSink.commitCommand and serverRuntimeStartup.enqueueCommand waited up to 1,346 s.
Between 02:02 and 02:41, ws.rpc.orchestration.dispatchCommand timed out 13 times after 180 s. Every agent wake-up in that window failed.
We killed the reader at 03:00:57. Dispatch calls returned in 1 to 12 s right after that, compared with 20 to 180 s or a timeout before.

This matches the diagnosis in the issue description: the checkpoint cannot pass the reader's snapshot, blocked stays true, and runWithAdmission waits without a deadline. We have not tested #23496 on this server.

Workaround we use until a fix ships: a small watcher outside T3 that never opens the database. Every 2 minutes it checks the size of statev2.sqlite-wal. If the file stays above 32 MiB for 5 minutes, it lists the processes that hold the database files open, read from /proc/*/fd. That way the stuck reader shows up within minutes, not after more than an hour.

Hitting this on a headless server too, and in my case the long reads mostly aren't outside processes. They're T3's own.

Setup: t3 serve 0.1.7 in Docker on Linux x64, statev2.sqlite at 18.7 GB, a bunch of Codex and Claude agents running overnight.

What I measured (about 5 hours of server.trace.ndjson):

25 reader transactions ran longer than 20 s. Nearly all were full shell loads (ProjectionStore.readShellSnapshotRows): about 16 s for the thread rows plus about 21 s for the per-thread item counts, every time. They come from the web app connecting, agents calling list_threads over MCP (one took 78 s), and a script of mine that reads /api/orchestration/shell on a timer.
62 writer transactions waited more than 3 s for admission (longest 24 s). 32 of them overlapped one of those >20 s reads, even though the long reads only covered about 5% of the window.
The WAL was at 34.9 MB when I checked, so past the 32 MB journal_size_limit.
Separately, I made a one-shot online backup of the DB yesterday (one read transaction, 145 s). A 55.9 s wait showed up right in the middle of it.

So any long read pins the WAL, including the server's own shell reads. Once the WAL is over the limit, every agent's writes queue behind whatever slow read happens to be running. From the UI it looks like the whole server froze.

I checked main @ 25bf84a, and gated() still includes blocked, so the nightlies have the same behavior. #23496 is 17 lines in nodeSqliteClient.ts and has independent verification above. Any chance it can get reviewed and merged? I'd rather not patch the bundled runtime myself. 🙏

Note

🤖 Claude Opus 5.5 posting on behalf of Etan, from his M1's traces.

Another data point, and it doesn't fit the long-reader explanation: a writer waited 25.5 minutes at checkpoint admission while no read transaction ran longer than 7.3 s.

Environment: macOS arm64 (M1), a busy install with many agent threads. The server was running 0.1.8-nightly.20261108.4353 (package metadata commit 3cd97a1e) when the traces were captured. This build includes #31804 (128 MiB WAL limit).

Stall 1 (trace 5328c973…), from server.trace.ndjson:

ProviderCommitBatch.flush                 1,532,243 ms
  EventSink.providerBatch                 1,532,243 ms   event_count=2
    sql.transaction (writer)              1,532,242 ms   permit wait 0.003 ms
      BEGIN                               1,532,230 ms   10:19:51.311Z -> 10:45:23.542Z
      ...nested work + COMMIT             ~12 ms at the very end (COMMIT 5.8 ms)

The batch took the in-process writer permit right away, then spent the whole 25.5 minutes in BEGIN. That's the runWithAdmission wait. Its SQL ran only after admission opened. Everything else queued behind the permit:

A user's message.dispatch waited 1,416,411 ms. Its own transaction took 20 ms once it got the writer.
Four thread.visit dispatches waited 1,024,584 / 950,872 / 917,281 / 712,825 ms (the Opening a thread (thread.visit) waits behind the single SQLite writer and shows "Some requests are slow" (34 s) #30301 symptom).
A waiting PRAGMA freelist_count recorded db.lock_wait_ms = 1,502,807, with the owner attributed to EventSink.providerBatch.

Stall 2, the same morning: EventSink.commitCommand waited 486,893 ms in BEGIN (10:54:47.950Z -> 11:02:54.844Z, db.lock_wait_ms = 486,871), then committed in 7.7 ms. A peer MCP call forwarded to this server timed out at its 300 s limit during this window.

What was not happening:

No reader sql.transaction longer than 20 s in either window. The longest in the captured traces (07:54–11:06Z) was 7,269 ms.
The event loop stayed responsive. reportClientActivity spans during stall 1 completed in about 0.1 ms, so this is not SQLite writes run on the server's event loop, so one slow transaction disconnects every client #30065.
The -wal file was 63.6 MB when checked shortly after stall 2. We have no WAL size or frame count from during the stalls.

What we couldn't determine: nothing in the trace records the gate state, i.e. which of unknown / blocked / pending truncate / frame ceiling held admission. Nor does it show what the checkpoint worker reported, or what reopened admission at 10:45:23Z.

Ruled out locally: a fixture with the real checkpoint worker, plus short readers that overlap every TRUNCATE attempt (each new reader opens before the previous one commits), does not stall on main @ 8e6a7323. The next TRUNCATE succeeds once the reader that predates the backfill is gone. So continuous short reads alone don't explain this. #23496 also passes that fixture.

Two things that would have answered this from the trace alone:

A gate-reason attribute on the admission wait.
The worker's last log / checkpointed / blocked result recorded on the waiting span.

Happy to run a build with those against this install, or to test #23496 on it.

Evidence gathered by Claude Opus 5.5 and Codex (gpt-6.1-sol / gpt-6-luna) via T3 Code, from the user's local traces. Identifiers and paths are trimmed.

Navigation Menu

Repository navigation

Any read transaction left open on statev2.sqlite freezes every server write once the WAL passes 32 MB #23482

What happened

Diagnosis

Steps to reproduce

Version

Environment

Evidence

Related issues

Fix applied or workaround

Filed by

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Any read transaction left open on statev2.sqlite freezes every server write once the WAL passes 32 MB #23482

Description

What happened

Diagnosis

Steps to reproduce

Version

Environment

Evidence

Related issues

Fix applied or workaround

Filed by

Activity

realhasanshoaib commented on Oct 22, 2026

aryaxo commented on Oct 22, 2026

realhasanshoaib commented on Oct 23, 2026

xdotli commented on Oct 25, 2026

grasdal commented on Oct 27, 2026

luworks commented on Oct 28, 2026

phkoenig commented on Nov 4, 2026

Sawtaytoes commented on Nov 8, 2026

EtanHey commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions