Repository navigation
Call-graph store is rebuilt from scratch on every engine start; refreshes on a 66k-file root read 176 GB in two days #746
Description
Version: AFT 0.62.0 (@cortexkit/aft-opencode) on OpenCode 1.19.1, WSL2 Ubuntu on NVMe, about 15 OpenCode instances (one AFT engine each).
On this machine the call graph turned out to be the largest disk consumer, while agents call aft_callgraph about 50 times a month. Three things add up:
1. Every engine start builds a new generation from scratch. Each start logs a full build (build_id=b-<pid>-1) and writes a new <key>.g<gen>.<pid>.sqlite, even when the previous generation of the root is intact. From our logs:
- a 66,189-file root:
elapsed_ms=531659; - an 8,505-file root:
elapsed_ms=103522. That engine's/proc/<pid>/ioshowed 15.6 GB written in its first 10 minutes, while the session only ran smallgit/grepcommands; - the same roots are rebuilt after every restart: one 3,871-file root on 2026-10-21 and again on 2026-10-22; another 2,651 files, then 3,596 files on the next start.
Since files already stores content_hash and mtime_ns, I expected a start to reuse the last published generation and re-parse only changed files.
2. Refreshes on a large store read and write a lot. Summing AFT's own legacy_callgraph_refresh events for the 66k-file root (890 refreshes, 2026-10-20T19:09Z to 2026-10-22T14:48Z): total_physical_bytes_written 16.6 GB, total_bytes_read 175.8 GB. The largest single refresh wrote 1.41 GB. About 200 MB read per refresh looks like whole-table work per refresh.
3. No backoff on writer_lease_timeout. When two engines served the same root (a setup bug on our side, now fixed), tier-2 builds failed with reason=writer_lease_timeout:_inspect_writer_lease_deadline_elapsed 628 times per category (dead_code, unused_exports, duplicates, cycles, complexity) over about 13 hours, roughly once a minute each.
Note on the 66k-file root: 50,067 of its files were sibling git worktrees, each a near-identical copy of the same WordPress tree. A warning when a root has many near-duplicate trees might help too.
Workaround here: "indexes": { "callgraph": false } for all instances. aft_callgraph then returns a clean callgraph_off, and aft_search and aft_zoom keep working.
Activity
mason: characterize call-graph store reuse across engine restarts and…
mason: clarify comments in the call-graph reuse and refresh I/O chara…
mason: reduce legacy callgraph build and refresh I/O (cortexkit#746)
mason: reconcile reused callgraphs after offline edits (#746)
mason: back off inspect lease retries and watch release (#746)
mason: explain callgraph cold-build decisions (#746)
mason: serve reused graphs during startup reconciliation (#746)
mason: make callgraph startup reconciliation stat-first (#746)
aft-alfonso commentedon Oct 24, 2026
Thanks for the detailed numbers. They made each problem easy to reproduce. Here is what we found and what changed on
main. All of it ships in the next release, and this issue stays open until that release is published.1. Rebuilds after a start. A plain restart already reuses the last published store, on 0.62.0 and on
main. The rebuilds in your logs had three other causes:main;There was also a real gap: edits made while no engine was running were never picked up after a restart. Now a start compares each file's size, modification time and file identity with the stored values, and re-parses only the files that changed. On a 60,000-file project, an unchanged restart now takes about 0.8 s and reads no file contents. While that check runs, answers come from the previous graph and are marked as possibly out of date. The log also records why each full build happened.
2. Refresh and build I/O. A one-file refresh used to read data in proportion to the whole store. On a 20,000-file test project it now reads about 2.5 MB instead of about 50 MB. A full build writes about 4 times less (2.1 GB instead of 8.6 GB on the same project). The store format changed, so each existing store rebuilds once after the upgrade.
3. Lease retries. When the inspect writer lease is busy, retries now back off from 30 seconds up to 30 minutes instead of retrying every 30 seconds.
Clones and worktrees. Clones of one repository still share one store in the current layout. The per-checkout index views we are moving to give each checkout its own graph. Thanks also for the note on near-duplicate worktrees inside one project folder; we are looking at how to surface that.
"indexes": { "callgraph": false }stays a supported setting if you prefer to keep it off.aft-alfonso commentedon Oct 27, 2026
Thanks for the report. AFT now skips a linked worktree of the same repository when it sits inside the project folder. Text search, semantic indexing, grep, glob, the call graph, inspect and the file watcher all leave it out, so its files are no longer indexed twice. Separate repositories and submodules inside the folder are still indexed, and a path you name directly inside such a worktree is still read.
The change is on main and ships in the next release. This issue closes when that release is published.
Merge pull request cortexkit#1010 from cortexkit/land/skip-nested-wor…