Snapshot repo bloats to 15 GB in 11.5 h on v2.0.27 (latest): 334k loose objects + tmp_obj_*/tmp_pack_* garbage; maintenance gc ineffective #55010

Activity

1. opencode-agent commented on Oct 13, 2026

   

   opencode-agentBot

   on Oct 13, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed numbers, they help a lot.

   One note first: v2 doesn't run the hourly git gc --prune=7.days that #17753 and #14811 refer to (that was v1). In v2.0.27, after a snapshot capture, OpenCode checks at most every 10 minutes and packs loose objects once there are 2048 or more. If packing fails, it logs a warning: failed to pack snapshot objects. Having 334k loose objects means that packing either failed again and again or never ran. To find out which, could you share:

   1. Logs. Any failed to pack snapshot objects or failed to capture snapshot lines from that session (OpenCode's log files), with the error text that follows them. If there are many, one or two examples is enough.
   2. The workload, in your own words, briefly. What was the agent doing for those 11.5 hours? For example: running builds or tests that write output into the project, generating or rewriting many files, several sessions or subagents running at once, or the same repository opened from several worktrees. Is the project a git repository, and are large directories (node_modules, build output) ignored in .gitignore?
   3. If you still have it or can reproduce it: the full git count-objects -vH output, and roughly how many tmp_obj_* and tmp_pack_* files there were.

   You can reply here or edit the issue. If you edit it, please also leave a short comment saying you've updated it, so we know you're done.

   Related issues, ordered from most to least likely to be relevant (none looks like the same problem):

   • v2: define snapshot retention and disk GC policy #36093: v2 snapshot retention and disk GC policy (snapshot objects are currently never pruned in v2)
   • Snapshot cleanup fails forever and leaks tmp_pack_* after project gc prunes shared objects #46034: v1 snapshot cleanup failing permanently and leaking tmp_pack_*
   • Periodic snapshot-maintenance git gc fails 100% from first tick in some sessions, while event-driven git spawns in the same session succeed #43877: v1 periodic snapshot gc failing on every run, so loose objects pile up
   • Snapshot system leaks tmp_pack_* files indefinitely, filling disk at ~9.5 GB/hour #14811 / Snapshot directory can grow to 40GB+ in 7 days due to hardcoded prune=7.days #17397: earlier v1 reports of the snapshot store filling the disk
2. opencode-agent

   added

   triaging

   on Oct 13, 2026
3. slamdunk111 commented on Oct 13, 2026

   

   slamdunk111

   on Oct 13, 2026

   Author

   More actions

   Thanks for the fast triage — here's everything requested.

   1. Logs

   From ~/.local/share/opencode/log/opencode.log (session run=95d0bef5, log starts 2026-10-12T15:54:21Z):

   timestamp=2026-10-13T01:21:24.548Z level=WARN run=95d0bef5 message="failed to pack snapshot objects" cause="Git.OperationError: fatal: sha1 file '/home/ubuntu/.local/share/opencode/snapshot/ce95.../62f3.../objects/pack/tmp_pack_odfHeH' write error. Out of diskspace" role=server
   timestamp=2026-10-13T01:21:24.794Z level=WARN run=95d0bef5 message="failed to capture snapshot" cause="Git.OperationError: fatal: unable to write loose object file: No space left on device" role=server
   timestamp=2026-10-13T01:21:39.333Z level=WARN run=95d0bef5 message="failed to capture snapshot" cause="Git.OperationError: fatal: unable to write loose object file: No space left on device" role=server
   

   Important nuance: failed to pack snapshot objects appears exactly once in the whole log, at 01:21Z — i.e. after the disk was already full. So packing was not "failing again and again" beforehand; it was logging no errors while loose objects still accumulated to 334k. The disk-full errors are a consequence, not the cause.

   2. The workload

   This is, I think, the key finding:

   • opencode was launched from /home/ubuntu — the home directory itself (log shows cwd=/home/ubuntu in 537 entries; the actual project being worked on was /opt/one-api-new, a small one-api deployment dir).
   • /home/ubuntu happens to be a git repository (2095 tracked files, 1888 dirty entries at last check, no .gitignore), and it contains .npm, .cache, .nvm, .local/share/opencode/opencode.db (+ -wal), go build caches, etc. — constantly-churning files totaling several GB.
   • The snapshot git dir lives inside the tracked worktree: worktree = /home/ubuntu, GIT_DIR = /home/ubuntu/.local/share/opencode/snapshot/<project-hash>/<repo-hash>/. If the capture does not exclude its own data directory, every capture commits the previous captures' loose objects and packs — a self-ingesting feedback loop. That would fully explain 15 GB / 11.5 h from a <1 GB project, and why packing "succeeded" (no warnings) yet loose objects exploded: each cycle re-commits the snapshot store's own churn.
   • Setup: one opencode serve --service process + interactive TUI, ~8 distinct run= ids over the 11.5 h window, single user, no worktrees, no parallel subagents that I know of.

   Could you confirm whether the v2 capture excludes the OpenCode data directory (or anything under ~/.local/share/opencode/) when it falls inside the tracked worktree? If not, that's likely the root mechanism here, and running opencode from $HOME is probably a common user pattern worth guarding against (e.g. always exclude the data dir, or refuse to snapshot $HOME).

   3. Object counts (captured live before cleanup)

   git count-objects -vH inside the snapshot repo:

   count: 334485
   size-pack: 6.68 GiB
   in-pack:  (large; repo total du = 15 GB)
   

   Garbage found by git gc (full warnings, before I interrupted it): 79 tmp_obj_* files (spread across objects/a7/, objects/b7/, objects/7a/, objects/7c/, objects/ae/, objects/b7/ …) and 1 tmp_pack_* file (objects/pack/tmp_pack_odfHeH).

   Note the tmp_obj_* garbage specifically: these sit under objects/xx/ as loose-object temp files, so a maintenance pass that only sweeps tmp_pack_* (per #14811) would never clean them.

   Recovery note

   I had to rm -rf ~/.local/share/opencode/snapshot to recover the host, so the repo itself is gone — the numbers above are from live captures during the incident. Workaround now in place: "snapshot": false in opencode.jsonc plus a weekly guard script (gc + size cap) as defense in depth.

   Happy to run a controlled repro (opencode from $HOME on a scratch VM with a disk quota, logging capture/pack cadence) if that would help pin the self-capture hypothesis down.
4. opencode-agent

   removed

   triaging

   on Oct 13, 2026
5. opencode-agent commented on Oct 13, 2026

   

   opencode-agentBot

   on Oct 13, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the details, your guess was right. I reproduced this on the latest dev (c1615b1). The snapshot code there is the same as in v2.0.27.

   When the project's git worktree contains OpenCode's data directory (here because $HOME is a git repo with no .gitignore), each snapshot capture also records the snapshot store's own files: its new loose objects, temporary index files and pack files under 2 MB. The script below recreates that layout and runs 12 captures, each changing only one line in one file:

   • The number of snapshot-store entries in each tree roughly doubles per capture (55 → 128 → 273 → 540 → 991 → 1661), and so do the loose objects.
   • After a pack, the new pack files are captured as well, and the growth starts again.
   • The store grows by about 1 MB per capture for a one-line change. In the app, packing runs at most every 10 minutes, so loose objects keep growing this way between packs. That fits the 334k loose objects you saw.

   The capture only filters by the project's ignore rules and the 2 MB limit on untracked files. Nothing excludes the snapshot directory itself. Separately, leftover tmp_obj_* files under objects/xx/ are never cleaned up; only tmp_pack_* files are.

   Until this is fixed, besides "snapshot": false you can add .local/share/opencode/ (or a broader rule, like .local/ and .cache/) to a .gitignore in your home repo, or start OpenCode from the actual project directory (/opt/one-api-new) instead of $HOME.

   Related: #16336 (v1, huge snapshot storage when started in the home directory) and #36093 (v2 snapshot retention policy).

   Reproduction script (selfcapture.test.ts)

   // Reproduces #55010: when the project worktree contains OpenCode's data dir (e.g. OpenCode run from a
   // $HOME that is a git repo without .gitignore), each snapshot capture re-ingests the snapshot store itself.
   // Mirrors packages/core/src/snapshot.ts: snapshot gitdir = <data>/snapshot/<project>/<hash>, seeded from the
   // source repo, captured with ignores=source and the 2 MiB untracked cap, then objects.pack.
   // Run from /workspace/repo:  bun test /workspace/repro/selfcapture.test.ts
   import { test, expect } from "bun:test"
   import { $ } from "bun"
   import fs from "fs/promises"
   import path from "path"
   import { Effect, Layer } from "effect"
   import { LayerNode } from "/workspace/repo/packages/util/src/effect/layer-node"
   import { Git } from "/workspace/repo/packages/core/src/git"
   import { AbsolutePath, RelativePath } from "/workspace/repo/packages/core/src/schema"
   
   const count = async (dir: string) => {
     let loose = 0
     for (const d of await fs.readdir(path.join(dir, "objects")).catch(() => [] as string[]))
       if (/^[0-9a-f]{2}$/.test(d)) loose += (await fs.readdir(path.join(dir, "objects", d))).length
     const du = (await $`du -sk ${dir}`.quiet().text()).split("\t")[0]
     return { loose, kb: Number(du) }
   }
   
   test("snapshot capture ingests its own store when the data dir is inside the worktree", async () => {
     const home = await fs.mkdtemp("/tmp/opencode/fakehome-")
     await $`git init -q ${home} && git -C ${home} config user.email a@b && git -C ${home} config user.name a`.quiet()
     await fs.writeFile(path.join(home, ".bashrc"), "echo hi\n")
     await $`git -C ${home} add .bashrc && git -C ${home} commit -qm init`.quiet()
     const gitDirectory = AbsolutePath.make(path.join(home, ".local/share/opencode/snapshot/proj/hash"))
   
     const program = Effect.gen(function* () {
       const git = yield* Git.Service
       const source = (yield* git.repo.discover(AbsolutePath.make(home)))!
       const repository = yield* git.repo.create({ worktree: AbsolutePath.make(home), gitDirectory, seed: source })
       const rows: string[] = []
       for (let i = 0; i < 12; i++) {
         // the user's actual change per step: one tiny file edit
         yield* Effect.promise(() => fs.writeFile(path.join(home, "work.txt"), `step ${i}\n`))
         const tree = yield* git.tree.capture({
           repository,
           scopes: [RelativePath.make(".")],
           ignores: source,
           maximumUntrackedFileBytes: 2 * 1024 * 1024,
         })
         yield* git.objects.pack(repository).pipe(Effect.ignore)
         const entries = (yield* Effect.promise(() =>
           $`git --git-dir ${gitDirectory} ls-tree -r --name-only ${tree}`.quiet().text(),
         )).split("\n").filter(Boolean)
         const self = entries.filter((e) => e.startsWith(".local/share/opencode/snapshot/")).length
         const c = yield* Effect.promise(() => count(gitDirectory))
         rows.push(`capture ${i}: tree entries=${entries.length} snapshot-store entries=${self} loose=${c.loose} store=${c.kb}KB`)
       }
       return rows
     })
     const rows = await Effect.runPromise(
       program.pipe(Effect.provide(LayerNode.compile(Git.node) as Layer.Layer<Git.Service>), Effect.scoped),
     )
     console.log(rows.join("\n"))
     expect(rows.at(-1)).not.toMatch(/snapshot-store entries=[1-9]/)
   }, 120_000)

   Run it: bun selfcapture.test.ts
6. opencode-agent

   added

   reproduced

   on Oct 13, 2026
7. slamdunk111 commented on Oct 13, 2026

   

   slamdunk111

   on Oct 13, 2026

   Author

   More actions

   Great — glad the self-capture hypothesis held up, and thanks for the reproduction script and the extra workarounds.

   Status on my side:

   • "snapshot": false + a weekly guard script (gc + size cap on the snapshot dir) are already deployed on the affected host, so the acute risk is covered.
   • I've also added .local/, .cache/, .npm/ and .nvm/ to a .gitignore in the $HOME repo as belt-and-braces. One caveat we noticed: ~231 files under those dirs were already tracked in the home repo, and ignore rules don't apply to tracked files — so this only stops new churn; fully de-tracking them would need git rm --cached, which I've left alone for now.

   Two follow-up questions, no urgency:

   1. Is a fix planned on your side — e.g. always excluding the OpenCode data directory from capture when it falls inside the worktree (independent of the project's ignore rules), plus sweeping tmp_obj_* in the maintenance pass? If so, I'll keep this issue open as the tracking reference for my notes.
   2. For v2: define snapshot retention and disk GC policy #36093 (v2 retention policy) — is the plan there to add a configurable size cap on the snapshot store? That would have made this incident self-limiting regardless of the self-capture bug.

   Happy to re-test on a scratch VM once a fix lands.