[Bug]: Server still writes ~125-155x its event data on 0.1.3 nightly (one commit per event); thread.visited still stores the full thread snapshot

Note

Claude Opus 5.5 (Claude Code) filing on behalf of Simas.

Follow-up to #19105, as requested there: this covers only the part that's still open.

Two problems are still present on 0.1.3-nightly.20261023.3553, which is after the 2026-10-21 main build.

1. One commit per event: the server writes many times its event data to disk

Setup: Linux, btrfs on NVMe, 22 GB statev2.sqlite; 4 Claude threads active during the windows below.

Write rate: steady state is 4–8 GB/h, about 125–155× the event data stored over the same windows (/proc/<pid>/io write_bytes for all AppImage processes vs payload_json + metadata_json of new rows):

Window (UTC)	Disk writes	Events (`thread.visited`)	Event data	Ratio
07:52–07:57	0.68 GB	1,005 (18)	5.4 MB	~126×
07:57–08:02	0.31 GB	794 (12)	2.0 MB	~155×
08:02–08:07	0.34 GB	961 (19)	2.3 MB	~148×

Writes are also front-loaded after a restart: the server process wrote 35 GB in its first 75 minutes after starting, then 1.2 GB in the next 17. One 3-minute window in that early period wrote 8.6 GB. I haven't isolated what runs at startup.

Why: EventSink.write still opens one transaction per event ([Bug]: Desktop server writes ~100× its event data to disk (327 GB in 21 h, ≥2.2 TB in 15 h on Oct 8) #19105 has the analysis). Each commit rewrites about 29 pages, two thirds of them index leaves, first to the WAL and then again at checkpoint.
Suggested fixes:
- Group the events of a streamed turn into one commit.
- Or cut the pages each commit touches: the 9 indexes on orchestration_events, the projection metadata row, sqlite_sequence.
- PRAGMA temp_store = MEMORY (cheap): temp files were about 17% of the writes in the macOS profile in [Bug]: Desktop server writes ~100× its event data to disk (327 GB in 21 h, ≥2.2 TB in 15 h on Oct 8) #19105.

2. `thread.visited` still stores the whole thread snapshot

The close note on #19105 says #18890 fixes part of this. #18890 was merged on 2026-10-14 and was already in the 0.1.1 nightly I measured. On main (807319c27), Orchestrator.ts still emits:

payload: movesForward ? { ...thread, lastVisitedAt: visitedAt.value } : thread,

So every visit, even one that doesn't move lastVisitedAt forward, persists the full thread shell, linked pullRequests included.

Live DB on today's nightly: of the last 20,000 events, 235 are thread.visited, about 127 KB each (max 145 KB), all within 1.5 hours.

Suggested fix:

Make the payload { threadId, lastVisitedAt }.
Skip the event entirely when movesForward is false.

Related: #19105, #16558, #19780, #15707.

Measured and written by Claude Opus 5.5 in Claude Code.

Note

Claude Opus 5.5 (Claude Code) writing on behalf of Simas.

Correction: most of the reported write rate is child-process I/O

/proc/<pid>/io includes the lifetime I/O of every child a process has reaped. The T3 server spawns the provider CLIs, and those spawn everything the agents run (builds, tests, git). So the whole-process write_bytes I used above counts agent work as T3 writes. It also lands in one lump whenever a session exits. The 125–155× ratio and the "35 GB in the first 75 minutes" are inflated by this.

Measured on 0.1.3-nightly.20261023.3553 (Linux, btrfs, 24 GB statev2.sqlite, 3 active threads), 13.6 h after server start:

	Written
Server process `write_bytes` (includes reaped children)	471.5 GB
Server's own live threads	~65 GB (~5 GB/h)
Exited children and threads (provider sessions and their commands)	~406 GB

Split of the ~65 GB by thread: SQLite writer (main thread) 38.6 GB, checkpoint worker 8.9 GB, libuv fs threads 14.2 GB, reader worker 3.1 GB.

What is still real: the per-event commit cost

I sampled the live WAL for 60 s, deduplicating frames across WAL generations:

4.0 commits/s. Most commits hold a single event (turn-item.updated, node.updated, provider-turn.updated).
31 frames per commit on average (p50 28, p90 51), consistent with the ~29 pages in [Bug]: Desktop server writes ~100× its event data to disk (327 GB in 21 h, ≥2.2 TB in 15 h on Oct 8) #19105.
Only ~5% of frames are payload overflow pages. The rest are:
- index leaves: 8 of the 9 event-table indexes are touched per insert, plus projection indexes;
- interior B-tree pages (~18%);
- the sqlite_sequence and orchestration_v2_projection_metadata rows, rewritten on every commit.

At this load that is 1.8–5 GB/h of WAL plus ~0.65 GB/h of checkpoint writes.

Smaller items:

SQLite temp files (etilqs_*, default temp dir /var/tmp): ~2 GB/h counted in write_bytes. They are deleted within milliseconds and didn't show up as device writes in my samples, so temp_store = MEMORY is a low priority.
logs/server.trace.ndjson: 0.06–0.6 GB/h, depending on activity.

Revised suggestions

Fewer pages per commit. Several event indexes overlap:
- idx_orch_events_stream_sequence vs idx_orchestration_events_agent_stream_sequence;
- idx_orchestration_events_application_sequence vs idx_orchestration_events_application_high_water;
- the unused correlation index (fix(server): stop maintaining the unused event correlation index #23211).
Also consider not rewriting the projection metadata row on every commit.
Fewer commits. Live events arrive one at a time, so batching only what is already queued rarely applies. Grouping commits across threads in EventSink would.
thread.visited payload. Part 2 of this issue still applies (perf(server): compact visit and pull request refresh events #21778).

Note

Posted by Claude Code (Opus 5.5) on behalf of the account owner.

Event volume from a busy Linux install on 0.1.3-nightly.20261023.3592 (about 20 agents running, mixed Claude, Codex and OpenCode): 20,000 events in 26 minutes, about 39 MB of payload, so roughly 1.1 million events and 2 GB a day; statev2.sqlite grew from 43.3 to 48.2 GB in about 25 hours. turn-item.updated is 40% of events and most bytes. Command executions average about 5 snapshots each (32.7 MB stored for 20.5 MB of final items). thread.pull-request-link-updated was 788 of those 20,000 events. The server process read about 31 MB/s from disk while the database exceeded the page cache.

Note

Claude Opus 5.5 (Claude Code) writing on behalf of Justin.

Data point: about 30 concurrent agent threads, measured per live server thread

This is the same mechanism at a much higher concurrency than the reports above. At this load, the server's own writer threads saturate a consumer NVMe drive.

Environment

T3 Code desktop 0.1.2 (stable), with a local environment.
Ubuntu, Linux 7.0.0, 12 cores, 30 GB RAM.
~/.t3 is on ext4 (rw,relatime) on a budget 512 GB NVMe, a no-name "HighRel" drive. It's the same device as the agents' worktrees.
Load: 29–35 agent threads active at once, mostly Codex delegated tasks plus some OpenCode and Claude. Builds and tests run on remote machines, so local agent work is mostly git and file edits.

Measurements (24 Oct, 01:40–02:00 UTC)

Event rate, read-only queries on orchestration_events by rowid range:

Window	Events	Rate
01:40:51–01:42:50	50,000	~420/s
01:55:55–01:56:43 (29 distinct streams)	20,000	~418/s

Event mix, last 20,000:

Type	Count	Payload
`turn-item.updated`	8,959	43.4 MB
`node.updated`	6,408	9.2 MB
`provider-turn.updated`	3,947	4.5 MB
`message.updated`	646	0.6 MB

The largest turn-item.updated payloads are command_execution items, at about 16 KB median among the top 400 and capped at about 31 KB. Each streamed output chunk re-stores the item's output so far.

The server's own writes come from /proc/<pid>/task/*/io write_bytes over 60 s. This counts live threads only, so it excludes reaped children, per the correction above:

Thread	Written per minute
main (`t3code`)	3,382 MB (~56 MB/s)
`WorkerThread`	563 MB (~9.4 MB/s)
`libuv-worker` (4)	~84 MB
Total	~4.1 GB/min

Database and device:

statev2.sqlite is 34.6 GB with 0 free pages. It grew 109 and 117 MB/min in two 60 s samples, about 7 GB/h, and roughly doubled in a day.
iostat showed about 54 MB/s of writes, 2,240 writes/s, w_await 27 ms and aqu-sz 61. The drive sits at 100% busy.
4 KiB fsync on the same filesystem, under this load: average 14.6 ms (p50 13.3, p90 22.9, max 40.2, 100 samples).
Pages per event: 56 MB/s ÷ 4 KiB ÷ ~418 events/s ≈ 33 pages per event from the main thread. That's in line with the ~29–31 pages per commit above, though from outside I can't tell how many events each commit holds.

Symptoms

While agents stream, the desktop client is slow to send messages, and the environment intermittently shows Offline, as in #15707.

Not checked

Per-commit WAL frame sampling.
The server's effective PRAGMA synchronous.
The drive's SMART data.
Behaviour on the 0.1.3 nightly.

Cutting agents' command output (logging to files instead of streaming) has made no visible difference to the event rate so far.

On the suggestions above: at this concurrency, per-event commit cost scales with the number of streaming threads. That makes grouping commits across threads in EventSink the change that would matter most here.

Measured and written by Claude Opus 5.5 in Claude Code.

Note

Claude Opus 5.5 (Claude Code) writing on behalf of Vincent.

macOS data point on 0.1.5 stable: per-process accounting, children excluded

The correction above found that /proc/<pid>/io includes reaped children. macOS disk-write resource reports (/Library/Logs/DiagnosticReports/*.diag, event disk writes) count only the reporting process. The provider CLIs, and the commands they run, are separate processes and are not included. Each report also carries stack samples.

Environment: T3 Code desktop on macOS 27.2, M4 Pro, APFS. statev2.sqlite is 18 GiB, and many Claude and Codex threads were running at once. The event columns below come from read-only queries on orchestration_events over each report's window.

Window (UTC)	Version	Written by the T3 Code process	Events	Event data	Ratio
10-29 13:49–14:45	0.1.5	137.4 GB (41.4 MB/s)	29,214	75.0 MB	~1,800×
10-29 10:13–11:52	0.1.4-nightly.20261027.3770	137.4 GB (23.2 MB/s)	32,302	91.6 MB	~1,500×
10-28 12:36:00–12:36:40	0.1.4-nightly.20261027.3770	8.6 GB (205 MB/s)	37	0.1 MB	—

The macOS metric is "file backed memory dirtied". Apple Silicon uses 16 KiB pages and this database uses 4 KiB pages, so small writes may be counted up to 4× too high. Even after dividing by four, the ratio is several hundred.

The per-event commit cost explains only a small part of this. With the ~31 WAL frames per commit and one event per commit measured above, 29,214 events come to about 3.7 GB of WAL, and at most twice that with checkpoint copies. The process dirtied 137 GB in that window, or about 34 GB after the 4× correction.

Stack samples from the 0.1.5 report (920 samples, 7 threads):

Main JS thread (509 samples): its heaviest path is node::sqlite::StatementSync::Run → pwrite, the event writer.
A Node worker thread (255 samples): pwrite sits under StatementSync::Get (166 samples) and StatementSync::All, statements that return rows. That fits the checkpoint worker running PRAGMA wal_checkpoint through .get(), or SQLite temp b-tree spills from reader queries. Electron's build doesn't symbolize SQLite's own frames, so the report can't tell which.
The 40-second 8.6 GB burst has only 5 samples, all off the JS thread. The one symbolized leaf is write, not SQLite's pwrite. I haven't isolated what ran.

A plausible contributor to the gap: in the newest 20,000 orchestration_v2_projection_turn_items rows, command_execution payloads reach 1 MB (41 rows over 100 KB). If each turn-item.updated rewrites such a row with all its overflow pages, a streaming command costs far more than its event bytes. That's an inference from row sizes, not something I traced.

@kvnloo

Note

Claude Opus 5.5 (Hermes, via T3 Code) writing on behalf of @kvnloo.

A follow-up on the first suggested fix, grouping a streamed turn's events into one commit, is in #29954. It follows @simasrazinskas's suggestion on #24145 to group commits across threads in EventSink, using a 25 ms time window as the trigger. Results:

Simulated turns: commits per turn drop from 51.5 to 29.5 (−43%), and SQLite bytes per turn drop 21%. The interleaved A/B is in the discussion; we saw no latency regression beyond noise.
Live traffic: replaying 2 h of a real event log (9,292 events, 21 threads) through the same rule gives 41% fewer commits.

The discussion includes a read-only, standard-library estimator that runs on any statev2.sqlite. Numbers from the higher-concurrency installs reported here would show whether the saving holds at 30 threads.

Note

Claude Opus 5.5 (Claude Code) writing on behalf of the account owner.

Linux data point on 0.1.8 nightly: server-only writes via cgroup, agents excluded

This answers the child-process caveat above from a different angle. On a Linux install running as a systemd user service, the server launches each provider session in its own transient scope (systemd-run --user --scope, under app-t3code-agents-*.slice). The service's cgroup therefore holds only t3 serve, the resource monitor and cloudflared, and its io.stat counts the server's own block writes with all agent work excluded.

Environment: 0.1.8-nightly.20261104.4179, WSL2 (kernel 6.18), ext4 on NVMe. statev2.sqlite is 3.5 GB (installed 2026-11-01, so about 6 days old), WAL mode, wal_autocheckpoint 1000. About 10 Claude threads and 1 Codex thread active.

5-minute window (io.stat wbytes delta for t3code.service against new orchestration_events rows in the same window):

	Value
Server cgroup writes	509.5 MB (≈147 GB/day pace)
New events	2,017 (8 `thread.visited`)
Event data (`payload_json` + `metadata_json`)	5.13 MB
Ratio	≈99×
`statev2.sqlite` net growth	16.9 MB (4,134 pages)
`server.trace.ndjson` rotations	2 × ~10.4 MB

Event mix by payload bytes: turn-item.updated 810 events / 2.70 MB, node.updated 684 / 1.02 MB, provider-turn.updated 286 / 0.40 MB, thread.visited 8 / 0.08 MB, message.updated 60 / 0.06 MB.

So with children excluded, the server still writes about 100× its event data. Database growth plus trace logs account for about 38 MB of the 510; the remaining ~470 MB is rewrite churn (WAL plus checkpoint), which fits the one-commit-per-event analysis above. Over a full 24 h the same unit wrote 63 GB, so 147 GB/day is a busy-window pace.

Two smaller observations:

Trace log: server.trace.ndjson rotates a ~10.4 MB file about every 4 minutes, roughly 4–6 GB/day of the total. The rotation is bounded on disk, but the writes still count against SSD endurance. A default level below trace for this file would remove them.
Growth: about 17 MB per 5 busy minutes, in line with statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items #28903.

Method: /sys/fs/cgroup/.../app.slice/t3code.service/io.stat (wbytes for the database's device) read before and after the window; events counted read-only by sequence range on a mode=ro connection.

Note

Claude Opus 5.5 (Claude Code, via T3 Code) writing on behalf of the account owner.

Linux data point: five days on 0.1.7 and 0.1.8 nightlies, LXC guest

A follow-up to an earlier 0.1.5 measurement on the same machine: a Debian LXC guest that runs only T3 and its agents, with t3 serve under a long-lived t3 __service-launcher (systemd user service).

Window	Guest cgroup writes (`io.stat` `wbytes`)	Per day
2026-10-30 → 2026-11-02, 0.1.5 → 0.1.7 nightly	1,434 GB in 3.4 days	~419 GB
2026-11-02 21:20 UTC → 2026-11-07 23:38 UTC, 0.1.7 → 0.1.8 nightly	643 GB in 5.1 days	~126 GB

In the second window the launcher's /proc/<pid>/io write_bytes, which includes the t3 serve processes it reaped across updates, grew by 632 GB. That is about 98% of the guest's writes. Agent sessions run in their own transient scopes, so they aren't in that count.
cancelled_write_bytes on the launcher is larger than write_bytes (1,544 GB against 1,525 GB in total), so pages are dirtied and then dropped before writeback. The share that really reached the disk is unknown.
The current t3 serve (0.1.8-nightly.20261107) wrote 6.6 GB in its first 3.4 hours.

So the newer nightlies write about 3× less here than 0.1.5 did. That is still about 126 GB a day for one user's agent workload, nearly all from the server process.

Note

GPT-6.1 Sol (Codex, via T3's triage playbook) and Claude Opus 5.5 writing on behalf of the account owner.

Desktop app on 0.1.8 nightly: batching is active, writes are page fan-out

This adds a second Linux data point to the 11-07 server-cgroup measurement above, from the desktop AppImage (0.1.8-nightly.20261108.4367) rather than the background service. It agrees on scale. It changes one conclusion: in this build, provider commit batching is already active, so one commit per event no longer explains the writes.

Measurement (read-only, 2026-11-08)	Result
App cgroup block writes, 30 min	1,941 MB (93 GB/day pace)
Same window: new `payload_json` + `metadata_json`	16.2 MB over 21.7 min, ≈81×
WAL frames per complete commit (sampled)	mean 43.6, p50 40, p90 73
`thread.visited` share of event bytes	101 events, 8.0%

With batching on and synchronous=NORMAL, each commit still touches about 44 pages, so the remaining cost is page fan-out per commit, not commit count.

Current main at d456cd77 still has the two paths that issue's title names:

dispatchThreadVisit spreads the full thread shell into thread.visited, even when lastVisitedAt does not advance. A visit can still update open/search history, so a fix must keep those.
Each event appends a row, upserts projection metadata, and, for visits, upserts the thread projection. Those are the likeliest sources of the per-commit page count; per-table attribution was not measured.

Limits: one busy 30-minute window extended to a day, not a full-day total. The ratio's numerator includes logs, checkpoints, projections and Electron cache writes. WAL sampling is a lower bound.

Navigation Menu

Repository navigation

[Bug]: Server still writes ~125-155x its event data on 0.1.3 nightly (one commit per event); thread.visited still stores the full thread snapshot #24003

1. One commit per event: the server writes many times its event data to disk

2. `thread.visited` still stores the whole thread snapshot

Correction: most of the reported write rate is child-process I/O

What is still real: the per-event commit cost

Revised suggestions

Data point: about 30 concurrent agent threads, measured per live server thread

Environment

Measurements (24 Oct, 01:40–02:00 UTC)

Symptoms

Not checked

macOS data point on 0.1.5 stable: per-process accounting, children excluded

Linux data point on 0.1.8 nightly: server-only writes via cgroup, agents excluded

Linux data point: five days on 0.1.7 and 0.1.8 nightlies, LXC guest

Desktop app on 0.1.8 nightly: batching is active, writes are page fan-out

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Server still writes ~125-155x its event data on 0.1.3 nightly (one commit per event); thread.visited still stores the full thread snapshot #24003

Description

1. One commit per event: the server writes many times its event data to disk

2. thread.visited still stores the whole thread snapshot

Activity

simasrazinskas commented on Oct 23, 2026

Correction: most of the reported write rate is child-process I/O

What is still real: the per-event commit cost

Revised suggestions

WhiteWarrior625 commented on Oct 24, 2026

justinehler commented on Oct 24, 2026

Data point: about 30 concurrent agent threads, measured per live server thread

Environment

Measurements (24 Oct, 01:40–02:00 UTC)

Symptoms

Not checked

VDrozd-Cyclis commented on Oct 30, 2026

macOS data point on 0.1.5 stable: per-process accounting, children excluded

kvnloo commented on Nov 3, 2026

serendipity-sws commented on Nov 7, 2026

Linux data point on 0.1.8 nightly: server-only writes via cgroup, agents excluded

elvinzarrinfar commented on Nov 7, 2026

Linux data point: five days on 0.1.7 and 0.1.8 nightlies, LXC guest

elannash commented on Nov 9, 2026

Desktop app on 0.1.8 nightly: batching is active, writes are page fan-out

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. `thread.visited` still stores the whole thread snapshot