[Bug]: Reopening a thread with a large event backlog freezes the UI while it visibly re-replays (quadratic replay) #4596

Activity

1. colonelpanic8 commented on Jul 26, 2026

   

   colonelpanic8

   on Jul 26, 2026

   ContributorAuthor

   More actions

   thread-replay-chug-4x.mp4

   Here is a video replay at 4x showing the issue with a reproduction thing that I built
2. colonelpanic8

   changed the title [-]Reopening a thread with a streaming-delta backlog freezes the UI for minutes (quadratic replay)[/-] [+]Reopening a thread with a large event backlog freezes the UI while it visibly re-replays (quadratic replay)[/+] on Jul 27, 2026
3. colonelpanic8 commented on Jul 27, 2026

   

   colonelpanic8

   on Jul 27, 2026

   ContributorAuthor

   More actions

   Update after validating against a real database: this is not streaming-gated. A sampled real state.sqlite (with enableAssistantStreaming off) has a thread carrying 4,830 events — 4,170 of them thread.activity-appended (tool calls). Replaying that exact mix through the playground (/perf.html?deltas=350&activities=4200&auto=1) chugs for ~37s with 7,114 React commits and a 568ms worst stall; batch=32 drops it to ~5.4s / 325 commits. The activity path is also the worst per-event reducer cost: threadReducer.ts:507-513 re-sorts the whole activities array on every event. Title and body updated accordingly; the mechanism and suspected fixes are unchanged.
4. colonelpanic8

   changed the title [-]Reopening a thread with a large event backlog freezes the UI while it visibly re-replays (quadratic replay)[/-] [+][Bug]: Reopening a thread with a large event backlog freezes the UI while it visibly re-replays (quadratic replay)[/+] on Jul 27, 2026
5. colonelpanic8 commented on Jul 27, 2026

   

   colonelpanic8

   on Jul 27, 2026

   ContributorAuthor

   More actions

   thread-replay-activities-2x.mp4

   not the best video because it doesnt follow the scroll but you can see how long it takes to play those back with the wall clock time.
6. colonelpanic8

   added 6 commits that reference this issue on Jul 27, 2026

   fix(client-runtime): batch thread stream application to keep resume r…

   ...

   a2a47ca

   fix(server): cap thread resume replay gap with a snapshot fallback

   ...

   b82f11e

   fix(client-runtime): batch thread stream application to keep resume r…

   ...

   0002cb2

   fix(server): cap thread resume replay gap with a snapshot fallback

   ...

   04cb294
7. chountalas commented on Jul 29, 2026

   

   chountalas

   on Jul 29, 2026

   More actions

   Adding a server-side profile, plus a structural trigger that I think explains why this fires far more often than "reopening a thread" suggests.

   Desktop nightly 0.0.30-nightly.20260728.935, macOS 26.5 on an M2 Max, Codex provider. Assistant text was arriving at roughly 4 tokens/sec. The same model driven from a plain terminal was fast at that same moment, and over the same 8 minute window codex app-server used 10s of CPU while the T3 backend child used 6m41s, pegged at 100% of one core. The cost was on our side.

   sample <backend-pid> 5, main thread, 3,443 samples:

   • 3,027 (88%) inside a single uv__run_check / CheckImmediate callback chain
   • 1,927 of those in node::sqlite::StatementSync::All
   • roughly 1,095 more in the same chain directly above that frame, which lines up with the per row Schema.decodeUnknownEffect plus two fromJsonString fields in OrchestrationEventStore.ts:31-33
   • only 416 samples (12%) anywhere else

   Database at capture: 904 MB, 92 threads, 260,679 thread events. 195,437 were thread.message-sent totaling 60 MB, and 57,181 were thread.activity-appended totaling 248 MB of payload. Three consecutive message-sent payloads:

   {"threadId":"…","messageId":"…","text":"",           "turnId":"…","streaming":…}
   {"threadId":"…","messageId":"…","text":".",          "turnId":"…","streaming":…}
   {"threadId":"…","messageId":"…","text":" throughout","turnId":"…","streaming":…}
   

   One event per streamed delta, averaging 319 bytes to carry a few bytes of text. Daily volume over five days of ordinary use, 24th through 28th of July: 3,504, then 22,575, then 64,655, then 60,811, then 103,082, with the last day still in progress when I captured this.

   Why the catch-up read specifically? There are only three steady state readers of orchestration_events. The projection pipeline sits at the head and reads tiny ranges. Shell replay is capped at SHELL_RESUME_MAX_GAP = 1000 with a snapshot fallback. Thread catch-up is readEvents(afterSequence, Number.MAX_SAFE_INTEGER) with the per thread filter applied after the read. That last one is the only unguarded large-read path on the table.

   The part I did not expect is that it recurs without reopening anything. In packages/client-runtime/src/state/threads.ts, applyItem advances lastSequence only to events arriving on this thread's stream (line 206), and the server stream is already filtered to that thread. So a thread's resume cursor lags the global head by the sum of every other thread's traffic, permanently, and the gap only widens. That same subscription resubscribes on every foreground wake and retries failures every 250ms (lines 293-297). At 100k events/day, full width global rescans recur on ordinary app focus, per cached thread state, with no reopen involved. It also explains two things I could not otherwise account for: restarting the app did not help at all, and a brand new empty thread was just as slow.

   On wording, I do not think "blocks" is right. readFromSequence pages at READ_PAGE_SIZE = 500 and the fiber scheduler yields between effects, so this is saturation rather than a monolithic block. The backend's single JS thread spends its time on synchronous SQLite reads plus per row decode, and everything else on that thread contends with it. NodeSqliteClient.ts is synchronous node:sqlite on the main thread behind a one permit semaphore, and there are no worker threads in apps/server, so that work has nowhere else to go.

   One control worth reporting. I purged only the thread rows and kept every project registered, including one pointed at a very large directory tree. Backend CPU dropped from 100% to 2.4% immediately. That seems to rule out VCS status scanning as the primary cost, at least here.

   This is the same mechanism the commenter on #996 described from a 4.43 GiB database, down to the stale subscribeThread cursors and the filtering after decode. Their prewarmer observation fits too. Opening ten threads at once froze the app completely here, which is what you would expect from ten unbounded catch-ups queued behind one semaphore.

   So, #3510. It already implements this, in its own words replaying at most 1,000 global events after the client cursor and serving a fresh thread snapshot beyond that. It has been open since June. It is large and touches 35 files, but its author notes the first server commit carries the reliability fix alone with no UI changes. From a user's side, landing just that piece would resolve everything I measured. The client already handles a kind:"snapshot" frame mid-stream and resets lastSequence when it does (threads.ts:195-199), so the fallback should not break resuming clients.

   Happy to put up a small server-only PR for the gap check and snapshot fallback if that would help, though I would rather not duplicate #3510 without a maintainer saying it is wanted.

   ---

   Workaround, and it costs you every thread you have. Conversations, messages, activities, all gone. Projects, sessions and paired devices survive. Back up first, and only do this if you are genuinely stuck.

   Quit the app fully, copy state.sqlite plus any -wal and -shm files somewhere safe, then delete orchestration_events where aggregate_kind = 'thread', the thread command receipts and the thread projection tables, and VACUUM. Keeping projection_projects, the auth tables, effect_sql_migrations and projection_state is what preserves projects and pairings. Here that took the database from 904 MB to 312 KB, and backend CPU from 100% to about 2%. Migrations do not re-run, and the AUTOINCREMENT counter stays level with the projector cursors, so nothing replays. Schema references are as of this nightly and may drift.
8. VincentShipsIt commented on Jul 30, 2026

   

   VincentShipsIt

   on Jul 30, 2026

   More actions

   Corroborating desktop profile (macOS 0.0.31) — agentic tool mix, not only stream-replay

   Adding a real local database snapshot that lines up with the “tool activities dominate” update on this issue.

   At capture

   • state.sqlite 587 MB
   • 151,749 orchestration events
   • 127,380 thread.message-sent (streaming was on)
   • 22,813 thread.activity-appended (~174 MB event payload / ~168 MB projected)
   • Active QA thread: 2,461 activities / ~80 MB projected payload alone
   • Another active QA: 981 activities / ~20 MB
   • Soft-deleted threads still retained full event logs (one archived thread had 102k events / 14.7k activities)

   Symptom
   After a few tool rounds on Grok Build in T3, the same turn becomes very slow. New empty threads are snappy. Matches the superlinear client cost story: each activity-appended grows the set that every subsequent apply re-sorts / re-derives.

   Also hit the catch-up side
   Server traces showed heavy subscribeThread + projector/SQL work even when not deliberately reopening cold tabs — consistent with other comments that this isn’t only “reopen after 5 min idle.”

   Recovery
   Purged deleted-thread data + wiped history on fat active threads + disabled enableAssistantStreaming + VACUUM → 288 KB DB. App usable again.

   Related signal for maintainers: #4008 (per-delta shell-summary / full activity rescan on the server path) stacked on top of this client quadratic apply. Fixing only one side still leaves the other.

   I can share a sanitized event-type histogram or a minimal perf.html comparison against this activity mix if that helps prioritize coalescing vs snapshot fallback vs activity paging.
9. juliusmarminge commented on Sep 15, 2026

   

   juliusmarminge

   on Sep 15, 2026

   Member

   More actions

   Closing as fixed by #8309 (and the batch apply already on main from #11302).

   The quadratic reopen/replay path this issue described is what the client-runtime batching + settled-turn/older-page fixes targeted; GitHub didn’t auto-close because the merged PR body had no Closes keywords.
10. juliusmarminge

    closed this as completedon Sep 15, 2026