server: git polling loop saturates CPU on large worktrees (snapshot dir inside project triggers rescan storm) #62948

Activity

1. opencodejr commented on Nov 9, 2026

   

   opencodejrBot

   on Nov 9, 2026

   More actions

   Thanks for the detailed report and the log excerpts. I tried this on 2.2.6 (Linux) but couldn't reproduce the polling loop.

   What I tried:

   • A HOME that is a git repo with 150k untracked, un-ignored files, with OpenCode's data folder (~/.local/share/opencode) inside it, and a mock model that streams each reply slowly over about 60 seconds.
   • opencode serve left idle: no git processes at all in 45 seconds.
   • One 60-second turn with opencode run: one snapshot scan when the turn started and one when it ended (diff-files + ls-files + update-index + write-tree, then one diff). Nothing ran afterwards while idle.
   • The TUI on a 40k-file project with snapshots on, kept open for about 3 minutes: again, scans only at the start and end of the turn.

   So in my runs each scan came from a turn starting or ending, not from a timer or from file-watcher events. I could confirm the other part of your report: when $HOME is the git worktree, the snapshot store records OpenCode's own data folder. That is already tracked in #55010, and the open PR #56209 would exclude that folder.

   To track down where your repeated scans come from, it would help to know:

   1. Whether the bursts also happen with no session running (no prompt in progress, no subagents), for example right after restarting the service.
   2. Whether they stop when you temporarily disable your plugins (opencode-models-discovery and ./koboldcpp-plugin), since a plugin that runs prompts or writes files could start new turns.
   3. A short log excerpt around one burst (say 20–30 lines before and after a group of ls-files spawns), so the lines that trigger it are visible, not just the git commands. Lines that mention a session or step start would be especially useful.

   Related issues:

   • Snapshot repo bloats to 15 GB in 11.5 h on v2.0.27 (latest): 334k loose objects + tmp_obj_*/tmp_pack_* garbage; maintenance gc ineffective #55010: the snapshot store records its own folder when $HOME is a git repo (the 31 GB growth you saw). PR fix(core): keep OpenCode data out of filesystem snapshots #56209 would fix it.
   • OpenCode falls back to user home directory (C:\Users\<user>) as root and scans the entire drive / massive Git repo #60370: a single snapshot scan is very slow on a huge home-folder worktree. PR perf(core): scan snapshot changes with one git status #59423 would make that scan cheaper.

   Until then, a .gitignore in $HOME that covers .local/ and other large data folders, or running OpenCode from a project folder instead of $HOME, avoids the cost of these scans.
2. Lapin-myope commented on Nov 9, 2026

   

   Lapin-myope

   on Nov 9, 2026

   Author

   More actions

   Thanks for looking into this — here are answers to your three questions, all from ~/.local/share/opencode/log/opencode.log (v2.2.6, log spans 2026-10-29 → 2026-11-09, 124,846 lines) plus a controlled A/B test I ran.

   1. Do bursts happen with no session running?

   Mostly no — an idle server does not poll. Controlled test: I started two fresh opencode serve instances (ports 4997/4998) in fresh 3000-file git repos under /tmp/opencode/repro{1,2} — one with OPENCODE_CONFIG={} (no plugins), one with the full user config (plugins enabled) — and left them with no client connected for ~90 s. Result: each spawned git exactly once at startup (workspace init), then silence (ls-files=1 each, no further spawns).

   Caveats from the production log:

   • The managed service (opencode serve --service, which chdirs to $HOME) does an initial scan right at startup: today's first ls-files spawn was at 14:41:20.592Z, ~17 s after the service process started and before the first HTTP event-stream connection (14:41:37.825Z GET /api/event).
   • Night bursts exist (885 ls-files spawns between 02:00–06:00 on 2026-11-06), but they were all cwd=/home/docker/project and there were cli starting events at 02:44:49 and 03:01:10 — i.e. clients were connecting at those hours. So bursts correlate with client connections / workspace (re)initialization, not with a blind timer.

   2. Do bursts stop with plugins disabled?

   Plugins are a strong trigger/amplifier while the server is active, but not the only one.

   • In the log, bursts of provider.updated / model.updated events (emitted by opencode-models-discovery@latest polling the configured providers — koboldcpp-plugin has refreshMs: 60000, modelsDiscovery enabled on the provider-a/provider-b providers) are immediately interleaved with git spawns (see excerpt below: event at 28.943Z → git spawn at 28.947Z). Each event batch seems to trigger a full git workspace re-discovery (symbolic-ref HEAD, remote, for-each-ref refs/heads, config init.defaultBranch, rev-parse) plus ls-files/diff-files status scans.
   • A/B test with plugins disabled (OPENCODE_CONFIG={}): no difference when idle (1 spawn each) — so plugins alone don't explain the loop. The second trigger is the file watcher: watcher subscribe path=~ type=entries ignores=0 — the entire $HOME (47 GB, ~1.25M files) is watched with ignores=0 (no ignore rules applied), and opencode itself writes constantly inside the watched work-tree: ~/.local/share/opencode/opencode.db-wal, snapshot/*/objects/*, shell/*/*.out, .git/objects/*, plus the log itself. Each of those writes is a watcher event → rescan. This feedback loop is independent of plugins and sessions.
   • I could not run "connected client + plugins disabled" without restarting the managed service (which would kill active sessions); the mechanism above is visible directly in the logs.

   3. Log excerpt around one burst

   Window 2026-11-09T19:49:28.805Z → 19:49:29.186Z (28 lines, 16 plugin events + 6 git spawns interleaved; full excerpt also saved at burst-excerpt-anon.log on the reporter's machine):

   19:49:28.805Z message=event event.type=model.updated    location.directory=/home/docker/project
   19:49:28.816Z message=event event.type=provider.updated location.directory=/home/docker/other-project
   19:49:28.842Z message=event event.type=provider.updated location.directory=~
   19:49:28.855Z message=event event.type=model.updated    location.directory=~
   19:49:28.928Z message=event event.type=provider.updated location.directory=~
   19:49:28.943Z message=event event.type=model.updated    location.directory=~
   19:49:28.947Z spawning process command=git args=[... "symbolic-ref","--quiet","--short","HEAD"] cwd=~
   19:49:28.955Z spawning process command=git args=[... "remote"] cwd=~
   19:49:29.014Z message=event event.type=provider.updated location.directory=~
   19:49:29.027Z message=event event.type=model.updated    location.directory=~
   19:49:29.030Z spawning process command=git args=[... "for-each-ref","--format=%(refname:lstrip=2)","refs/heads"] cwd=~
   19:49:29.044Z spawning process command=git args=[... "config","init.defaultBranch"] cwd=~
   19:49:29.048Z message=event event.type=agent.updated    location.directory=~
   19:49:29.098Z spawning process command=git args=["rev-parse","--git-dir","--git-common-dir","--show-toplevel"] cwd=~
   19:49:29.104Z watcher subscribe path=~/.git/HEAD type=file ignores=0
   19:49:29.104Z watcher started   path=~/.git/HEAD type=file backend=node ignores=0
   19:49:29.186Z spawning process command=git args=["rev-parse",...] cwd=~
   

   Adjacent lines in the same burst also contain the heavy scans against the snapshot repo: git --git-dir ~/.local/share/opencode/snapshot/<hash>/<id> --work-tree <project> ls-files --others --exclude-standard -z, diff-files --name-only -z, diff --name-status/--numstat/--unified=2147483647, check-ignore --stdin -z.

   Other relevant counts in the log: 22,370 ls-files spawns total; dense clusters e.g. 277 spawns within ~1,300 log lines; watcher subscribe path=~ type=entries ignores=0 (whole home tree, no ignores); 276 failed to capture snapshot and 196 failed to diff session turn (snapshot capture keeps failing/retrying against the huge tree).

   Summary of the mechanism: client connect or plugin event → workspace (re)discovery + status scan via git; on a 1.25M-file work-tree each scan takes seconds of >100% CPU and saturated disk; opencode's own data dir (~/.local/share/opencode: db, snapshots, shell output, logs) lives inside the watched tree with ignores=0, so its own writes re-trigger scans → self-sustaining storm while any client is connected.

   burst-excerpt-anon.log
3. opencodejr commented on Nov 9, 2026

   

   opencodejrBot

   on Nov 9, 2026

   More actions

   Thanks a lot for the detailed follow-up and the A/B test, it narrowed things down. I tried both triggers you pointed to on 2.2.6, but couldn't get the scans to repeat:

   • File watcher: $HOME was a git repo with 150k untracked files and OpenCode's data folder inside it, with a client connected. The watcher did subscribe to $HOME with ignores=0, as in your log. But writing files into $HOME and into ~/.local/share/opencode for 20 seconds started no git processes at all.
   • Provider/model updates: reloading the config repeatedly sent bursts of provider.updated / model.updated events (14–17 of each), both with only an event stream and with the TUI open on a session. Neither started any git processes: no rediscovery, and no ls-files / diff-files.
   • Snapshot scans: in every run they happened only when a turn started and ended.

   In your excerpt, the rediscovery lines (symbolic-ref, remote, for-each-ref, rev-parse, watcher on ~/.git/HEAD) look like the location being opened, which in my runs happened once, when a client first connected. So what keeps re-triggering it, and the heavy snapshot scans, on your side is still unclear. Two things would help:

   1. In burst-excerpt-anon.log, the nearest lines before the first ls-files/diff-files of a burst that mention a session, step, prompt or location, e.g. session, step, prompt, instance or location messages, or an HTTP request line. That would show whether each burst comes from a turn (perhaps one started by a plugin or another client), from a session diff request, or from the location being opened again.
   2. A few of the failed to capture snapshot and failed to diff session turn lines with their cause= text. If captures keep failing on the huge tree, something may be retrying them, and that would be a concrete lead.

   Related: #55010 tracks the snapshot store recording its own folder when $HOME is a git repo (PR #56209 would exclude it), and #60370 covers how slow one scan is on a huge home folder (PR #59423 would make it cheaper).