Activity

1. opencode-agent commented on Oct 12, 2026

   

   opencode-agentBot

   on Oct 12, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report and measurements. I could reproduce this on 2.0.27, and it still happens on the latest dev (bcf23b6).

   What I did on Linux, with a fresh data directory:

   1. Left a ~500 MB un-checkpointed WAL with many freed pages next to the database (a sqlite3 writer killed before it could checkpoint).
   2. Made the disk slow by adding a 2 ms delay to every read and write with strace.
   3. Ran opencode service start.

   The log shows Background service is unresponsive about every 11 seconds. Each time the CLI kills the service that is still starting and spawns a new one. The WAL checkpoint never finishes, so the main database file never grows. On dev one of the extra services also failed with Managed service port ... is already in use by another process, the same error you saw. Without the slowdown, the same store starts in under a second.

   The script below reproduces it. I couldn't test the machine-wide freeze, since that needs a real HDD, but the kill-and-respawn loop that repeats the expensive startup work is confirmed.

   Related: #41739 (similar stall on a large database, caused there by the V1 migration), #54267 (port already in use while an older service is still alive), #54181 and #37795 (a busy service treated as dead by short health probes).

   Reproduction script (repro.sh)

   #!/usr/bin/env bash
   # Reproduces #54810: a slow-starting managed service (large WAL to checkpoint on a slow disk)
   # is declared "unresponsive" by the CLI after ~10s, killed and replaced, over and over,
   # so the WAL is never checkpointed.
   # Needs: opencode (or OC_DEV=/path/to/checkout), sqlite3, strace (slow disk is simulated by
   # delaying every pread64/pwrite64 by 2 ms).
   set -u
   ROOT=/tmp/opencode/repro54810
   rm -rf "$ROOT"; export HOME="$ROOT/home"; mkdir -p "$HOME"
   if [ -n "${OC_DEV:-}" ]; then
     cd "$OC_DEV/packages/cli"  # its bunfig.toml preload is needed by the spawned service child too
     OC=(bun --conditions=browser --preload="$OC_DEV/packages/tui/node_modules/@opentui/solid/scripts/preload.js" "$OC_DEV/packages/cli/src/index.ts")
   else
     OC=(opencode)
   fi
   "${OC[@]}" --version
   D="$HOME/.local/share/opencode"
   NAME=opencode; [ -n "${OC_DEV:-}" ] && NAME=opencode-local  # source builds use opencode-local.db/.log
   
   # 1. create the store, then stop the service
   timeout 60 "${OC[@]}" service start
   pkill -9 -f 'serve --service' ; sleep 1
   
   # 2. leave a ~500 MB WAL behind (writer killed before checkpoint), with many freed pages
   { echo "PRAGMA journal_mode=WAL; PRAGMA wal_autocheckpoint=0; CREATE TABLE junk(x BLOB);
   WITH RECURSIVE c(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM c WHERE i<250000) INSERT INTO junk SELECT randomblob(1000) FROM c;
   DELETE FROM junk WHERE rowid%4<>0; SELECT 'done';"; sleep 600; } | sqlite3 "$D/$NAME.db" > "$ROOT/sq.out" 2>&1 &
   for i in $(seq 120); do grep -q done "$ROOT/sq.out" 2>/dev/null && break; sleep 1; done
   pkill -9 -x sqlite3; sleep 1
   ls -la "$D" | grep db
   
   # 3. start the service on a simulated slow disk
   timeout -s KILL 90 strace -f -qq -o /dev/null -e trace=pread64,pwrite64 \
     -e inject=pread64:delay_exit=2000 -e inject=pwrite64:delay_exit=2000 \
     "${OC[@]}" service start > "$ROOT/start.out" 2>&1
   grep -v -e dispatch_event -e '^ *at ' "$ROOT/start.out"
   pkill -9 -f 'serve --service'; pkill -9 -x strace
   
   echo "--- log ---"
   grep -E 'service starting|unresponsive|cli process failed|serve\\",\\"--service' "$D/log/$NAME.log" | cut -c1-220
   ls -la "$D" | grep db
   echo "unresponsive-kill cycles: $(grep -c 'Background service is unresponsive' "$D/log/$NAME.log")"

   Run it: bash repro.sh
2. opencode-agent

   added

   reproduced

   on Oct 12, 2026
3. OhnkytaBlabdey commented on Oct 12, 2026

   

   OhnkytaBlabdey

   on Oct 12, 2026

   Author

   More actions

   Follow-up: offline repair results, a correction, and what actually writes to the store at startup

   1. Correction to the original report

   The original report suggested that a ~300 MB WAL had to be re-checkpointed on every launch. I
   have to withdraw that as a mechanism: on this machine the WAL file was 314,096,472 bytes but
   PRAGMA wal_checkpoint(TRUNCATE) returned 0|0|0, i.e. the file had simply never been
   truncated and there was nothing to say it held pending frames. The measured facts in the
   original report (duration, I/O volume, CPU, RSS, disk saturation) stand; the "300 MB
   re-checkpoint each launch" explanation does not, and the real writer is described in §3 below.

   2. Offline repair (store shrunk, startup no longer minutes)

   Performed with opencode fully closed, on the same machine:

   VACUUM INTO '<backup path>';          -- compacted copy, also serves as a backup
   PRAGMA wal_checkpoint(TRUNCATE);
   VACUUM;
   PRAGMA integrity_check;               -- ok

   Results:

   item	before	after
   opencode.db	5,239,611,392 B	2,056,183,808 B
   free pages (freelist_count)	917,638 pages	0
   opencode.db-wal	314,096,472 B	0 B
   integrity_check	—	ok
   sessions / messages / parts	984 / 38,409 / 245,697	984 / 38,409 / 245,697

   Total wall time for the three steps: 5 min 09 s (77 s for VACUUM INTO, 228 s for VACUUM).

   Effect on startup, same command line (opencode serve --hostname 127.0.0.1 --port <p>), same
   machine:

   	time from process start to server listening
   before repair, cold file cache	~5 min 59 s
   before repair, warm cache	~100 s
   after repair	10–12 s

   So the multi-minute startup is gone. This is on a machine whose only disk is a 4 TB SATA HDD.

   3. What still writes: startup populates the v2 tables

   The repair does not stop the heavy writes. In short guarded runs after the repair, the
   process still performed 0.7–1.4 GB of file I/O in the first ~1 minute, and the store kept
   growing even though the v1 row counts never changed.

   Measured before → after the first start → after a second start:

   table / file	after repair	after a 45 s start	after a 72 s start
   session	984	984	984
   message	38,409	38,409	38,409
   part	245,697 rows / 1,207,555,189 B	unchanged	unchanged
   session_v2	0	668	863
   session_message	0	24,637	34,906
   opencode.db size	1.96 GB	2.13 GB	2.45 GB
   freelist_count	0	0	0

   Observations:

   • The writes are new rows in session_v2 / session_message (the v2-shaped tables), populated
     from the existing v1 data (session / message / part), while part content is untouched.
   • The work is progressive: each process start advances it (session_v2 0 → 668 → 863 out of
     984 sessions) and costs ~170–320 MB of store growth and ~0.7–1.4 GB of I/O.
   • Because a start in this environment (HDD, large history) is heavy enough to make the machine
     unresponsive, it has been interrupted every time, so the backfill never finishes and every
     launch repeats a large chunk of work. The store had reached 5.24 GB with 72 % free pages
     this way.
   • The additional per-start I/O (0.7–1.4 GB for what is effectively "continue migrating my own
     history") is also what keeps the machine unusable during the first minute of a start, even
     after the store has been compacted.

   Questions:

   1. Is this v1 → v2 backfill expected to be chunked per process start like this, and is it
      resumable/idempotent? Can it be run to completion offline (a CLI command, or a documented
      "migrate and exit"), instead of depending on the server surviving long enough?
   2. During the backfill there is no log output at all between cli starting and the migration
      lines — could the backfill report progress so users don't have to guess whether it is hung?
   3. Should the backfill be rate-limited or deferred so that a first start after an update does
      not need to rewrite the whole store while the user is waiting for a TUI?

   4. Reproduction

   -- row counts before a start
   SELECT count(*) FROM session_v2, session_message;  -- (two queries: 0 and 0 here)
   -- start the server once and let it run for ~1 minute, then stop it
   SELECT count(*) FROM session_v2;      -- 668
   SELECT count(*) FROM session_message; -- 24637

   The same growth is visible as PRAGMA page_count increasing by tens of thousands of pages per
   start while PRAGMA freelist_count stays 0.

   5. Update: the backfill does finish, and after that the start is quiet

   A third start was allowed to run for 116 s while per-process I/O deltas were sampled every 5 s:

   measurement	value
   server listening	5 s
   total process I/O	read 300 MB / write 288 MB
   write delta after ~50 s	0 MB per 5 s sample, for the remaining ~70 s
   session_v2	863 → 984 (= number of sessions)
   session_message	34,906 → 39,347
   session / message / part	984 / 38,409 / 245,697 — unchanged
   opencode.db	2.45 GB → 2.58 GB, freelist_count 0

   So the run of three starts after the offline repair moved 0 → 668 → 863 → 984 sessions, and the
   third one finished and then went completely silent on writes.

   Practical consequence: the "first start after this migration" is a one-off heavy operation
   (~1–2 minutes of sustained I/O once the store is compact), and it has to be allowed to finish —
   interrupting it makes the next start repeat a chunk of it. Doing VACUUM +
   wal_checkpoint(TRUNCATE) offline before that first start shortens the waiting a lot (here
   server listening went from ~6 min to 5–12 s), but it does not remove the backfill itself.

   ---

   后续补充：离线修复结果、一处更正，以及启动时到底在写什么

   1. 对原报告的一处更正

   原文推测"每次启动都要重新 checkpoint 约 300 MB 的 WAL"。这条作为机制我要撤回：在这台机器
   上 WAL 文件确实是 314,096,472 字节，但 PRAGMA wal_checkpoint(TRUNCATE) 返回 0|0|0，说明它只
   是一个从未被 truncate 的文件，没有任何证据表明里面有待回拷的脏页。原文里实测的部分（耗时、
   I/O 体量、CPU、RSS、磁盘打满）仍然成立；"每次启动重做 300 MB checkpoint" 这个解释不成立，
   真正的写入方见第 3 节。

   2. 离线修复（库已收缩，启动不再是分钟级）

   在 opencode 完全关闭的情况下，于同一台机器上执行：

   VACUUM INTO '<backup path>';          -- 压缩式副本，同时作为备份
   PRAGMA wal_checkpoint(TRUNCATE);
   VACUUM;
   PRAGMA integrity_check;               -- ok

   结果：

   项目	修复前	修复后
   opencode.db	5,239,611,392 B	2,056,183,808 B
   空洞页（freelist_count）	917,638 页	0
   opencode.db-wal	314,096,472 B	0 B
   integrity_check	—	ok
   会话 / message / part	984 / 38,409 / 245,697	984 / 38,409 / 245,697

   三步合计耗时 5 分 09 秒（VACUUM INTO 77 秒，VACUUM 228 秒）。

   对启动耗时的影响（同一命令行 opencode serve --hostname 127.0.0.1 --port <p>、同一台机器）：

   	从进程启动到 server listening
   修复前、冷缓存	约 5 分 59 秒
   修复前、热缓存	约 100 秒
   修复后	10–12 秒

   也就是说，分钟级启动已经消失（该机器唯一的数据盘是一块 4 TB SATA 机械硬盘）。

   3. 仍然在写的是什么：启动时会填充 v2 表

   修复并没有消除重负荷写入。修复后几次受限运行中，进程在启动后第一分钟内仍然产生
   0.7–1.4 GB 文件 I/O，并且库在持续增长，而 v1 表的行数没有任何变化。

   实测（修复后 → 第一次启动后 → 第二次启动后）：

   表 / 文件	修复后	跑 45 秒后	跑 72 秒后
   session	984	984	984
   message	38,409	38,409	38,409
   part	245,697 行 / 1,207,555,189 B	不变	不变
   session_v2	0	668	863
   session_message	0	24,637	34,906
   opencode.db 体积	1.96 GB	2.13 GB	2.45 GB
   freelist_count	0	0	0

   观察：

   • 写入的是 session_v2 / session_message（v2 形态的表）里的新行，来源是已有的 v1 数据
     （session / message / part），而 part 的内容完全没动。
   • 这是渐进式的：每次启动都会往前推进（session_v2 0 → 668 → 863，共 984 个会话），
     单次代价约 170–320 MB 的库增长和 0.7–1.4 GB 的 I/O。
   • 在这种环境（机械盘 + 大量历史）下，启动本身就重到会让机器失去响应，于是每次都被中断，
     回填因此永远跑不完，下次启动再重复一大段工作。这个库就是这样涨到 5.24 GB、72% 是空洞页的。
   • 这份"每次启动额外 0.7–1.4 GB I/O、只为继续迁移自己的历史"的开销，也是库收缩之后、启动
     第一分钟内机器仍然不可用的原因。

   想请教：

   1. 这个 v1 → v2 回填是否本来就设计成"每个进程启动推进一块"？它是否可恢复/幂等？能否离线
      跑完（比如一个 CLI 命令，或"迁移完就退出"的模式），而不是依赖服务能活到跑完？
   2. 回填期间从 cli starting 到迁移日志之间完全没有输出——能否打印进度，让用户不必猜它是不是
      卡住了？
   3. 是否应该给回填限速或延后执行，使"升级后第一次启动"不必在用户等 TUI 的时候重写整个库？

   4. 复现

   -- 启动前统计行数（本例均为 0）
   SELECT count(*) FROM session_v2;
   SELECT count(*) FROM session_message;
   -- 启动服务并让它跑约 1 分钟，然后停掉
   SELECT count(*) FROM session_v2;      -- 668
   SELECT count(*) FROM session_message; -- 24637

   同样的增长也可以从 PRAGMA page_count 每次启动增加数万页、而 PRAGMA freelist_count 保持 0
   观察出来。

   5. 更新：这个回填确实会跑完，跑完之后启动就安静了

   允许第三次启动连续运行 116 秒，同时每 5 秒采样进程 I/O 增量：

   观测项	数值
   server listening	5 秒
   进程 I/O 总量	读 300 MB / 写 288 MB
   约 50 秒后的写入增量	每个 5 秒采样均为 0，持续其后约 70 秒
   session_v2	863 → 984（等于会话总数）
   session_message	34,906 → 39,347
   session / message / part	984 / 38,409 / 245,697 — 不变
   opencode.db	2.45 GB → 2.58 GB，freelist_count 为 0

   也就是说，离线修复之后的三次启动把它推成了 0 → 668 → 863 → 984 个会话，第三次跑完，随后写入
   完全静默。

   实际含义：这次迁移之后的"第一次启动"是一次性的重活（库收缩后仍需 1–2 分钟持续 I/O），必须让
   它跑完——中断它会让下次启动重复其中一段。在首次启动之前离线做 VACUUM +
   wal_checkpoint(TRUNCATE) 能大幅缩短等待（本例 server listening 从约 6 分钟降到 5–12 秒），
   但并不能省掉回填本身。
4. Dante-dan commented on Oct 12, 2026

   

   Dante-dan

   on Oct 12, 2026

   Contributor

   More actions

   I’m taking the managed-service startup retry fix. I’m checking the health-probe recovery against the existing slow-start and unresponsive-service contracts so a still-initializing child can finish within the startup window.
5. rorar commented on Oct 19, 2026

   

   rorar

   on Oct 19, 2026

   More actions

   Another data point for the same spawn storm, on Linux 2.0.32. Here the trigger was memory pressure, not a slow disk. Two details differ from the report above: the extra serve --service processes did not exit, and each new process made the stall worse.

   Environment

   • opencode 2.0.32 (latest channel). The service ran 2.0.32. The 4 TUI clients were still the 2.0.24 binary in memory, because the update replaced the binary under them.
   • Linux, inside an LXC container with cgroup v2 limits: memory.high 28 GiB, memory.max 32 GiB, no swap.
   • 4 long-running TUI clients in 4 projects (oldest 11 days), all sharing the default service on port 49374.

   What happened

   An unrelated process in the container slowly grew to 8 GB and pushed the cgroup above memory.high. The kernel then throttled and reclaimed continuously (memory.events high: 137 million, pgmajfault: 1.46 billion). Nothing was OOM-killed, so the container stayed in this state for hours.

   The service boot time (location services booted durationMs, max per hour, UTC) shows the onset:

   hour	max boot time
   2026-10-07 to 2026-10-18T18	0.1 to 1.3 s
   2026-10-18T20	5.8 s
   2026-10-19T01	31 s
   2026-10-19T02	376 s
   2026-10-19T04 onward	no boot completed

   At 2026-10-19T01:29Z the running service stalled. FileSystem.isDir and FileSystem.resolve calls ended in InterruptError: All fibers interrupted without error, requests ran for about 128 s and ended with status 499, and the service stopped its watchers. Within 10 seconds the 4 clients started 7 serve --service processes. From then on every client logged Timed out waiting for the background service to start and started another one every 3 to 10 minutes.

   Counts from opencode.log for 2026-10-19 04:00Z to 15:59Z:

   	count
   serve --service starts	232 (about 40 per hour)
   failed with Managed service port 49374 on 127.0.0.1 is already in use by another process	198
   answered at least one HTTP request	3
   logged only cli starting, nothing else	31

   The extra processes stayed alive

   When I stopped it, 66 serve --service processes were alive, 16 or 17 per client. The parent PID of each one was its TUI client. They were started over 5 hours. 12 were in state D, most of the others in state R. There were also [opencode] <defunct> zombies under the clients.

   Each process held only 35 to 90 MB resident, but under reclaim every one of them adds pressure and CPU time. So the storm feeds itself: the slower the service boots, the more contenders start, and the more contenders run, the slower everything gets. The host (12 cores) reached a load average of 202 with 91 % system CPU time, so other workloads on the same host stalled too.

   I cannot tell from the log whether the 66 survivors are contenders that hit the port error and then failed to exit, or contenders that hung before they reached the bind. The log has no exit line for either case.

   Without memory pressure the logic works. With a healthy service on 49374, a second serve --service exited at once and the first one kept the port.

   Recovery

   Killing the 4 clients, all serve --service processes and the memory hog ended it. SIGTERM was not enough under reclaim, so SIGKILL was necessary. The run queue dropped from about 160 to 3 to 8 runnable processes within 1 minute.

   Suggestions for the fix in #54883

   • Cap the number of contenders and back off exponentially. A fixed retry every few minutes per client never converges while the machine stays slow.
   • Let only one client start a contender at a time. Here 4 clients started 7 contenders in 10 seconds.
   • Make sure a contender that loses the port exits, and that the client reaps or kills the children it started.

   This comment was drafted together with Claude Code (Claude Opus 5.5). The numbers come from my opencode.log, from ps and from the cgroup files, and were checked on the affected machine before posting.
6. neriousy commented on Oct 23, 2026

   

   neriousy

   on Oct 23, 2026

   Member

   More actions

   Preserving the additional reproduction from #57038: on Windows, an nthLink network stall triggered the same managed-service recovery loop, with roughly 40 changing PIDs and repeated outbound requests. Please ensure the retry work in #54883 covers this network-stall trigger as well as slow startup.
7. neriousy

   marked service: recovery loop spawns unbounded serve processes when service becomes unresponsive #57038 as a duplicate of this issue on Oct 23, 2026