core: session never drains when Snapshot.capture() stalls on huge untracked tree — git add hangs, execution claim never released #48657

Activity

1. github-actions

   assigned

   neriousy

   on Sep 12, 2026
2. piknockyou commented on Sep 30, 2026

   

   piknockyou

   on Sep 30, 2026

   More actions

   Your analysis of the snapshot step matches what I hit, so I'm adding the second trigger in case it helps others landing here with the same symptom (no reply and/or the server restarted while you were working loop, event stream disconnected / stalled).

   In my case the huge tree was not a real project: a stray empty .git in a parent folder (my home folder) made opencode treat everything underneath it as one giant project. Same result you describe - the per-message scan takes ~26s for me (541k files), the watchdog gives up, the service gets restarted, and it loops.

   For anyone with this symptom: run git rev-parse --show-toplevel in the folder where you start opencode.

   • If it prints your home folder (or a big parent folder) instead of your actual project, a leftover .git is the culprit - delete it (usually empty, nothing lost; rename if unsure) and start opencode inside a real project folder. That was the fix for me.
   • If it prints your real project, then it is exactly this issue's case: get the heavy folders (node_modules/, target/, build/, caches) into .gitignore, or set "snapshot": false.
3. cdhcrs commented on Oct 9, 2026

   

   cdhcrs

   on Oct 9, 2026

   More actions

   Field data: 2.37M small files → I/O-bound stall, disk-fill risk, and no client-side recovery

   Hitting the same failure at a much larger scale, with several details I don't think are covered above.

   Environment

   • opencode version: 2.0.26 (latest channel)
   • OS: Linux 6.8.0-138-generic (Ubuntu 22.04, linux x64)
   • Shell: /bin/zsh
   • Active plugins: @openviking/opencode-plugin

   Scale

   • The session directory contained 2,376,791 non-gitignored files in a single dataset subdirectory (808 GB, avg ~350 KB/file). The repo itself tracks only 69 files.
   • The project .gitignore is the stock Python template; the dataset directory was never ignored.

   ---

   1. This instance is I/O-bound, not CPU-bound

   Unlike the git add pathspec matching described above (100% CPU), the failing command here is update-index, and it blocks in the kernel:

   PID 4166555  ELAPSED 19:43  STAT Ds  WCHAN folio_wait_bit_common
   git --git-dir <snapshot-git-dir> --work-tree <project-dir> \
       update-index --add --remove --replace -z --stdin
   

   D = uninterruptible disk sleep. It had been running 19 min 43 s when I killed it.

   2. Disk-fill risk

   update-index --add writes a loose object for every untracked file it ingests. Measured growth of ~/.local/share/opencode/snapshot/<project>/<hash>/:

   03:35:16  20857 MB
   03:35:26  20961 MB      (~10 MB/s)
   03:40:37  24471 MB
   

   At ~11 MB/s it would have tried to copy 808 GB into the snapshot object DB. Root filesystem had 637 GB free → roughly 17 hours to a full disk. The snapshot dir grew 18 GB → 24.5 GB during this session alone.

   3. The >2 MiB capture cap doesn't protect against many small files

   The snapshots doc says "Individual untracked files larger than 2 MiB → No". Every one of these 2.37M files is ~350 KB, so all are eligible. There is no file-count or total-size guard at all.

   4. No backoff on retry

   I kill -9'd the stalled process; OpenCode re-spawned the identical command within 60 seconds:

   03:40:21  WARN "failed to capture snapshot"
             cause="... Process interrupted due to receipt of signal: 'SIGKILL'"
   03:41:06  spawning process command=git
             args=["...","update-index","--add","--remove","--replace","-z","--stdin"]
   

   No failure threshold, no backoff, no "the previous attempt ran 20 minutes" check — a deterministic retry loop that keeps re-creating the condition.

   5. POST /interrupt returns interrupted: false, so there is no client-side recovery path

   While wedged:

   Call	Result
   GET /api/session/active	{"type":"running"}
   POST /api/session/{id}/interrupt	{"interrupted": false} — no interruptible fiber
   POST /api/session/{id}/revert/stage	60 s → 499 (InterruptError at SessionRevert.stage)
   DELETE /api/session/{id}	60 s → 499 (InterruptError at Session.remove)
   POST /api/experimental/session/{id}/wait	60 s → 499 (InterruptError at Session.wait)

   So the recovery path recommended in #44807 (POST /interrupt) does not work in this case: the execution claim is held by a live but blocked fiber, not a crashed run. All three escape hatches (interrupt / delete / revert) fail, and the sessions stay running.

   Both stuck sessions left running on their own only after I killed the git process and removed the stale index.lock — one transitioned to succeeded, the other to failed. POST /interrupt did not achieve this.

   6. Contradicts the documented "best effort" contract

   From the snapshots doc:

   Capture is best effort. If either snapshot is unavailable, the conversation continues, but that step may not have file changes available to restore.

   Here the unavailable snapshot doesn't degrade — it blocks every session operation in that project indefinitely.

   ---

   Reproduction

   1. In a git repo, place ~2M small (<2 MiB), non-gitignored files in one subdirectory.
   2. Open a session in that directory and send any prompt.
   3. Observed: no model call is ever made (process/stream never appear in the log), the session sits in active with {"type":"running"} indefinitely, and interrupt returns interrupted: false.

   Workaround verification

   After adding the dataset directory to the project .gitignore:

   git check-ignore -v <dataset-dir>
       → .gitignore:148:<dataset-dir>  <dataset-dir>
   git ls-files --others --exclude-standard | wc -l
       → 2376791  →  10
   time (git ls-files --others --exclude-standard >/dev/null)
       → 0.002s   (previously 39–45s)
   git ls-files | wc -l
       → 69 → 69   (tracked content unaffected)
   

   Suggestion

   Beyond the capture timeout / claim release already discussed here: a pre-flight guard that skips capture with a loud warning above N files or M bytes total, and treats a capture exceeding some duration as failed instead of letting it hold index.lock. Also worth revisiting whether a per-file size cap is the right axis when the pathological case is file count, not file size.
4. cyanyux commented on Oct 19, 2026

   

   cyanyux

   on Oct 19, 2026

   More actions

   Additional observation on OpenCode 2.0.32, Linux x64, accessed through T3 Code desktop connected to a Linux T3 server.

   A prompt asking only for progress spent several minutes in snapshot capture before its shell tool ran. Capture then failed on an unreadable file, and subsequent capture attempts caused further delays.

   Selected timestamps from October 19, 2026, UTC:

   12:49:36       Progress prompt submitted in T3
   12:50:12.300  Snapshot git update-index started
   12:58:46.557  failed to capture snapshot: Permission denied
   12:58:46.630  Progress-query shell tool started
   12:59:03.454  Shell tool completed, exit code 0
   12:59:08.900  Snapshot git update-index started again
   13:02:42.562  failed to capture snapshot: same permission error
   13:02:48.547  Another snapshot git update-index started
   

   The failing file was owned by root:root with mode 0600. Its path and contents are omitted. The active capture command was:

   git --git-dir <snapshot-git-dir> --work-tree <project-dir> \
     update-index --add --remove --replace -z --stdin
   

   During inspection, the Git process alternated between running and D state with folio_wait_bit_common; it had read approximately 4.1 GB from storage after about 65 seconds in that capture attempt. These process counters are cumulative for that attempt, not a measurement of the whole turn.

   At the time of inspection, the shell query had already returned its result, but T3 continued displaying the turn as working without a final response. This observation demonstrates repeated long capture delays; it does not establish a permanent stall or an infinite retry loop.

   This adds an unreadable-file failure case on 2.0.32 to the large-tree/update-index observations above. Early detection or exclusion of unreadable paths, bounded capture time, and a visible capture-failure status could help.

   No snapshot configuration, file permissions, or provider processes were changed during this investigation. Disabling snapshots has not been tested here.
5. Vijayapardhu commented on Oct 21, 2026

   

   Vijayapardhu

   on Oct 21, 2026

   More actions

   Adding Windows evidence for this same root cause — #57841.

   Three differences from the report here, in case they change the fix or the triage:

   1. Windows, and the legacy shell junctions are the amplifier. The working directory is $HOME, which is a git work-tree. $HOME also contains the legacy Windows junctions (Application Data, Local Settings, Cookies, My Documents, SendTo, Start Menu, Templates, Web, WebComponents), each of which re-exposes AppData as its own top-level tree. git therefore walks the same AppData content nine more times, and the untracked set that reaches the indexer is 1,126,526 paths — 6.7x the 167k in this report. AppData alone accounts for 1,039,918 files.

   2. The command changed between 2.0.2 and 2.1.2. On 2.1.2 the blocking call is no longer git add --all --sparse --pathspec-from-file=-:

   git --git-dir="~/.local/share/opencode/snapshot/global/<hash>" --work-tree="~" \
       update-index --add --remove --replace -z --stdin
   

   The untracked paths are still enumerated in full and streamed to a single indexer invocation, so the unbounded-pathset problem is unchanged in 2.1.2 — but the fix surface is update-index --stdin now, not git add --pathspec-from-file.

   3. The stall happens before any session row exists. With opencode run from $HOME on 2.1.2, no row is written to the session / session_v2 tables at all and the CLI emits zero stdout — so it is not obvious from the outside that the snapshot indexer is the thing blocking. In this report the user message is persisted but no assistant message follows.

   Quantified on the affected machine:

   	files	time
   git ls-files --others --exclude-standard	1,126,526	84.67s
   same, with a .gitignore covering AppData/, the junction dirs, tool state and caches	5,325	0.36s
   opencode run before the .gitignore	—	never completes (child git.exe at ~99% CPU, >207s CPU observed, index.lock held)
   opencode run after	—	7.1s

   The .gitignore workaround described in this issue is confirmed to work, and snapshot: false avoids the path entirely.

   Env: OpenCode 2.1.2 (channel latest), Windows 11 build 26200 x64, PowerShell 7.6.7, git for Windows 2.x. No plugins.

   One suggestion in case it is cheap: on Windows, treat the legacy junction directories as excluded, and/or skip the snapshot index when the working directory is $HOME. A bound or timeout on the untracked-path enumeration would also let a pathological tree degrade to snapshot: undefined (which capture() already supports) instead of blocking session startup.
6. Vijayapardhu commented on Oct 21, 2026

   

   Vijayapardhu

   on Oct 21, 2026

   More actions

   Correcting a detail in my comment above, and adding a portable reproduction with a threshold measurement.

   Correction. I wrote that the legacy Windows junctions in $HOME (Application Data, Local Settings, Cookies, …) re-expose AppData and inflate the path set ~9x. That is wrong. They are NTFS junctions, and git does not traverse them — git ls-files --others --exclude-standard -- "Application Data" returns 0 paths, same for Local Settings. AppData itself is 1,039,918 of the 1,126,526 paths (92%); the junctions add nothing. So please disregard the "9x" framing, and disregard the suggestion to treat them as excluded — that would not change the path count at all.

   Portable minimal reproduction (no large home directory required). Scratch git repo, 200,000 untracked empty files, OpenCode 2.1.2, Windows 11, stdin closed, same command and same provider for both runs:

   	untracked paths	opencode run wall clock
   bulk/ present	200,000	238.0s
   after adding bulk/ to .gitignore	1	4.6s

   A 52x difference with nothing else changed, and the provider is not contacted until the capture returns. This reproduces outside $HOME entirely, so it is independent of the home directory, of the junctions, and of the provider — it is a plain "large untracked tree in a git project" case, and it is a much cheaper repro than a 1.1M-file tree.

   This is a threshold datapoint for the cap/skip suggestion in this issue: ~200k untracked paths is enough to add ~4 minutes to every session start, and the cost scales roughly linearly with the untracked count. The machine in my original report reaches 1,126,526 paths (84.67s for ls-files alone) and update-index never completes — >207s CPU observed, still running after 5+ minutes, index.lock held in the snapshot repo — at which point opencode run does not return at all.

   Other facts confirmed on 2.1.2 / Windows 11, for the record:

   • the blocking call is git update-index --add --remove --replace -z --stdin (same form as @cyanyux's 2.0.32 comment above), so the fix surface is the same in 2.1.2;
   • it holds index.lock in the snapshot repo;
   • no provider request is made until it returns;
   • workarounds confirmed on Windows: a .gitignore covering AppData/, tool state and caches, or snapshot: false.

   Filed as #57841 (the bot triaged it as a duplicate of this issue — agreed, and it also relates to #50734 and #32981); cross-posting here so the measurements live with the fix.