Embed builds leak daemon RSS on ≥0.56.1 — remote backends always, local fastembed on 0.56.2

Short description

Full semantic builds accumulate daemon RSS without bound (0.5–3 GB/min) while the build itself progresses normally: every remote (openai_compatible) build on ≥0.56.1, and on v0.56.2 the local fastembed backend as well. v0.56.0 was memory-stable for hours on the same setup.

What happened?

During full semantic builds (~44,200 files / 375,627 chunks / 5,870 embed batches) the daemon's
RSS grows without bound while batches advance normally — no errors, no exceed_context events,
normal embed rate. The allocator slack stays roughly constant (~1.1–1.5 GB), so the growth is
live working set. Left alone, the daemon reaches host memory limits within 10–30 minutes of
embed-phase start and the plugin watchdog SIGKILLs it.

Three reproductions (RSS via /proc//status VmRSS plus allocator slack relief log lines):

Run A (v0.56.2, openai_compatible → llama.cpp llama-server, direct, no proxy, views off):
RSS 0.82 GB at min 3 → 16.2 GB at min 10 (~1.7 GB/min). Manually SIGTERMed before the watchdog.
Run B (v0.56.2, same remote setup): RSS 1.03 GB at min 42 → 10.9 GB at min 62 (~0.5 GB/min),
batch 680→1120 (43.5k→71.7k chunks). Manually SIGTERMed.
Run C (v0.56.2, local fastembed backend, same tree): RSS 17.4 GB at minute 20 of the embed
phase (daemon logged local embedder ready: model=all-MiniLM-L6-v2 intra_threads=6).
Manually SIGTERMed. The leak is therefore backend-independent on 0.56.2.

Version matrix from our runs:

v0.56.0 + identical remote backend and tree: 3h22m build, RSS steady at 1.4–1.9 GB — the
leak arrived with the 0.56.1-era embed-path rework.
v0.56.1 + remote: leaks (multiple runs).
v0.56.1 + local fastembed: one full 3h+ CPU build completed stably (RSS 2.4–2.6 GB band) —
on 0.56.1 the leak was remote-only.
v0.56.2: leaks on BOTH backends (runs A–C).

Isolated so far: no proxy in path, views disabled, max_input_tokens at both 448 and 512,
two different large repos, GPU-served and CPU-local embeddings — reproduces in every
combination on the affected versions.

Suspects from the diff: the ≥0.56.1 embed-loop rework — per-item EMA / adaptive batch growth
handling, retry/backoff bookkeeping, or response/batch buffering in the build embed pipeline.
The rate varies between runs (0.5–3 GB/min), suggesting accumulation tied to batch count or
content rather than wall-clock. Note: #319 documents another 0.56.0→0.56.1 regression in the
same embed path (query-time stall, fixed in 0.56.2) — this leak is from the same rework
window and survives 0.56.2.

Impact: at this corpus size no ≥0.56.1 backend configuration completes a build — remote leaks,
local leaks on 0.56.2, and downgrading to 0.56.0 remote reintroduces the oversize-row aborts
that #318 just fixed. (The watchdog SIGKILL also leaves stale daemon state that degrades
subsequent sessions — can file separately if useful.)

Diagnostics

`doctor --issue` could not run in the affected container (non-interactive terminal; the filing
flow requires a TTY). Manual context instead:

- Runs A/B: `semantic.backend = "openai_compatible"` → llama.cpp llama-server (all-MiniLM-L6-v2,
  `n_ctx_train=512`), direct connection, no proxy; run C: default local fastembed backend
  (bundled ONNX Runtime 1.24.4, intra_threads=6)
- Config: `max_files: 50000`, `max_input_tokens` tried at 448 and 512, `views.enabled: false`,
  bridge `request_timeout_ms: 600000`, `hang_threshold: 5`
- Host: OpenCode 1.18.31 (V1), plugin @cortexkit/aft-opencode 0.56.2, AFT binary 0.56.2
- Corpus: ~44,200 files / 375,627 chunks / 5,870 embed batches per full build
- RSS measured via `/proc/<pid>/status` (VmRSS) and `allocator slack relief` daemon log lines
  (`rss_bytes_before`); allocator slack held constant (~1.1–1.5 GB) in all runs

Plugin version

0.56.2

AFT binary version

0.56.2

Platform

linux x64 (daemon in a container; remote runs served by llama.cpp on the same docker network with Intel iGPU/Vulkan; local runs used bundled ONNX Runtime 1.24.4)

Log output (optional)

# Run A (v0.56.2, remote) — spawn, healthy first relief, then balloon, manual TERM
2026-09-21T07:07:39Z INFO [aft-plugin] Spawning binary: .../aft/bin/v0.56.2/aft
2026-09-21T07:10:40Z [aft] allocator slack relief: rss_bytes_before=Some(2230923264) rss_bytes_after=Some(823398400)
# ~07:18Z: VmRSS 16.2 GB via /proc
2026-09-21T07:18:35Z INFO [aft-plugin] Process exited: code=143, signal=null

# Run B (v0.56.2, remote) — healthy ~40 min, then balloon, manual TERM
# min 42: VmRSS 1.03 GB; min 62: VmRSS 10.9 GB (batch 680 -> 1120, no errors)
2026-09-21T09:09:53Z INFO [aft-plugin] Process exited: code=143, signal=null

# Run C (v0.56.2, local fastembed) — balloon on CPU
2026-09-21T09:42:19Z [aft] local embedder ready: model=all-MiniLM-L6-v2 intra_threads=6 ...
# ~10:03Z (min ~20): VmRSS 17.4 GB via /proc, stage=embed batch ~421, no errors
# manual SIGTERM; host recovered from 380 MB free to 23 GB available

# build_progress lines continued normally throughout all runs, e.g.:
#   kind=build_progress plane=semantic build_id=b-1306-1 stage=embed batch=421 total_batches=5870 ... (local)
#   kind=build_progress plane=semantic build_id=b-1514-2 stage=embed batch=1120 total_batches=5870 ... (remote)

Thank you for this — the version matrix and the flat allocator slack are what make it actionable, and the two-stage arrival (remote at 0.56.1, local added at 0.56.2) is the most useful fact in the report.

One correction to the suspect list, so nobody wastes a day on it: there are no commits touching the embed loop (crates/aft/src/semantic_index.rs) between v0.56.0 and v0.56.1 — 27 commits in that range, none in that file. So whatever made remote leak in 0.56.1 is somewhere else. The 0.56.1 → 0.56.2 embed changes are real and are the natural suspects for the local arrival (the context-overflow row recovery from #318 retains rows while it shrinks a batch, which is the right shape for a per-batch accumulator), but any candidate has to explain the 0.56.1 remote case too, and that one cannot be it.

Investigation is dispatched, bisecting the two arrivals separately against a reproduction driven by a stub embedding server, so batch count is the variable and the backend is not. RSS recorded against batch number rather than wall clock, per your observation that the rate tracks batches.

No workaround to offer you yet, and I would rather say that than guess one. Your own data already rules out the obvious knobs — max_input_tokens at 448 and 512, both backends, views off.

On the stale daemon state after a watchdog SIGKILL: please do file that separately. A SIGKILL mid-build is a state we should survive cleanly regardless of what caused it, and it is worth its own fix.

Progress report, and an honest one: we have not reproduced it yet.

We built a harness that records daemon RSS against batch number (rather than wall clock, per your observation that the rate tracks batches) and ran v0.56.0, v0.56.1, v0.56.2 and current main through ~470 batches each on macOS. All four came back flat at roughly 0.3 MB/batch, against the ~22 MB/batch your runs imply. Five different load shapes on main — a long quiet build, a deliberately slow backend, heavy concurrent tool traffic, repeated reconfigures, and the embed lane alone — were also flat. So whatever is happening is not reproducing in our environment, and I would rather tell you that than offer a guess dressed as a finding.

Your bisect stands, and we independently confirmed the part that matters: no commit between v0.56.0 and v0.56.1 touches the embed path at all, and the dependency set did not change either. So the 0.56.1 remote arrival cannot be an embed-loop change, which rules out the most natural-looking suspect and makes your two-stage matrix more interesting rather than less.

The next run targets your environment specifically — a Linux container, since several paths only exist there (the inotify watcher over tens of thousands of files, the cgroup CPU-quota reader added in 0.56.2, /proc memory accounting), and then the daemon driven through the subc bridge rather than the standalone transport, which is where your request_timeout_ms and hang_threshold settings live and the one layer our harness has not exercised.

Two things that would help, if you have them to hand:

Whether the daemon is running under a container memory limit or cgroup CPU quota, and what they are. The cgroup reader is one of the Linux-only paths and it arrived in the window you identified.

Whether a much smaller corpus on the same setup also grows — say a few thousand files rather than 44,200. If it does, that gives us a reproduction we can iterate on in minutes; if it does not, the accumulator is likely tied to absolute corpus size rather than batch count, which is a different search.

No workaround yet. Your own matrix already rules out the obvious knobs, and I will not suggest one we have not measured.

Two answers

1. Container limits / cgroup reader

No limits on the agent container that runs the daemon:

docker inspect ...agent-1: Memory=0 NanoCpus=0 CpuQuota=0 CpuShares=0
In-container /sys/fs/cgroup/memory.max = max (cgroup v2, unconstrained)
In-container /sys/fs/cgroup/cpu.max = max 100000 (no quota)
nproc = 12

The daemon's own cgroup reader fires at startup and reports its finding:

local embedder ready: model=all-MiniLM-L6-v2 intra_threads=6
  intra_threads_source=parallelism available_parallelism=12 cgroup_quota_threads=none

cgroup_quota_threads=none — the reader found no cgroup CPU quota and fell back to parallelism-derived thread count. So the leak runs in a fully unconstrained container; the daemon picked its thread count from raw core count (6 = 12/2, the same heuristic ONNX uses).

2. Small-corpus reproduction

I tried and did not get a clean small-corpus run completed within the timeframe — full transparency on what blocked me so you can decide whether to attempt it yourself (which would be ~5 min for you) or whether the inferential answer below is enough.

What I tried. Stood up a throwaway container from the same agent image (docker run --rm ... opencode with a 3,000-file synthetic Java corpus at /work, 12 MB total). The plugin-init flow kept hitting a Magic-Context / auto-compaction conflict dialog blocking the TUI; the aft plugin did not finish registering inside the dialog window, and the aft index CLI path (no TUI) refused to self-download the binary in this environment (aft index requires a native AFT binary; run aft doctor to install or repair it. even after a successful aft doctor --fix in the same shell — the index path has a separate resolution I couldn't satisfy non-interactively). I stopped rather than burn more time on plumbing. The throwaway container is gone; host memory fully reclaimed.

Inferential answer from the three repro runs you already have — and its limit. All three used the same 44,200-file corpus, and the leak rates differ substantially:

Run	Backend	Window (min)	RSS growth	Rate
A	remote	min 3 → min 10 (7 min)	+15.4 GB	~2.2 GB/min
B	remote	min 42 → min 62 (20 min)	+9.9 GB	~0.5 GB/min
C	local	min 0 → min 20 (20 min)	+17.0 GB	~0.85 GB/min

On closer look, the same rate spread is also consistent with host-load throttling of the embed loop — Run A happened during a busy session window (concurrent opencode sessions and a views test running in parallel); Run B later in the same session with less contention; Run C on a quieter CPU-only path. High host load slows the embed loop, so the same per-batch accumulation would show as a slower wall-clock rate. So the three runs cannot discriminate between corpus-size-independent accumulator throttled by host load and corpus-size-independent batch-rate dependence — both are consistent with the observed spread. A small-corpus test in a quiet window remains the cleanest discriminator.

Second round done, and it is another negative. We still cannot reproduce it, and I would rather keep telling you that than hand you a guess.

What we covered since the last update, all measuring RSS against batch number:

Linux container, remote lane, 938 batches, on v0.56.1, v0.56.2 and current main: 0.19-0.23 MB/batch, flat on all three.
Linux container, local fastembed lane, 625 batches, with a cgroup CPU quota actually applied and confirmed in the daemon's own log: 0.143 MB/batch, flat. This closed a real gap in our first round, which had only ever run the remote lane and so never instantiated the in-process ONNX embedder at all — your second arrival and the cgroup thread-count reader were both untested until now.
Your scale: 44,200 files, 353,600 chunks, 5,525 batches end to end: 0.205 MB/batch, flat. Peak 4.50 GB during persistence, settling to 2.43 GB.

Against your ~22 MB/batch, so whatever is happening needs something we have not modelled. One layer remains untested — the subc bridge transport, where your request_timeout_ms and hang_threshold live — and that is next.

The honest caveat on the scale run: we matched your file and batch counts with synthetic uniform Rust files, not your content. If the accumulator is content-shaped rather than count-shaped, that run would not see it.

Three things would move this further than more guessing on our side:

One leaking build with search_index, callgraph_store and views all disabled and no agent attached. This is the one that discriminates cleanly: if it still leaks, the accumulator is in the semantic build path and our harness is missing something specific about it; if it stops, the growth is in a plane running beside the build, which is a different search entirely.
/proc/<pid>/smaps_rollup at two points about twenty minutes apart during a leaking build. VmRSS cannot separate heap from mapped artifacts, and that distinction decides whether this is allocation at all.
The complete daemon log from a leaking run. Its index_event stream would tell us whether two semantic builds are ever live at once on your setup — the one hypothesis a synthetic corpus cannot rule out, and one that fits a rate varying between runs.

Correction to our earlier note: we had repeated your "0.56.1 + local fastembed completes" line in our investigation document as if it were tested. It was your report, not our finding, and the document now says so. A three-hour build is also slow enough that a leak might not have dominated within that window, so we are not treating it as a workaround.

My last comment crossed yours — I posted it without having read your 12:59 reply, so it asked for things you had already answered. Apologies; answering properly now.

The aft index failure is our bug, and it is the important thing in your comment. You did nothing wrong: doctor --fix downloads the binary into <cache>/v<version>/aft, and aft index resolves through a code path that only looks in that directory when it is given an explicit version — which it never is. So it checks the platform npm package, then PATH, then ~/.cargo/bin, and on a container with none of those it reports "requires a native AFT binary" pointing you at the command you had just run successfully. A fix is in flight, including making the message say where it looked, because a remedy the reader has already tried reads as "you did it wrong".

That also means the small-corpus run was blocked by us, not by you, and I am not going to ask you to fight our plumbing again to produce it. Once the fix ships, aft index on a 3,000-file tree in a quiet window should be a two-minute run.

The cgroup answer is more useful than it looks. cgroup_quota_threads=none with available_parallelism=12 is a different path from the one we just tested — our Linux round applied a real quota and confirmed cgroup_quota_threads=6. Both land on 6 threads so the derived value matches, but the branch does not, and "we tested the other side of a conditional" is exactly the kind of gap worth knowing about. Thank you for pulling the daemon's own line rather than just the container config.

Your throttling reading is right and it corrects mine. I had taken the 0.5-2.2 GB/min spread as evidence of batch-count dependence. Run A during a busy window, B quieter, C CPU-only is at least as good an explanation, and a slower embed loop would show identical per-batch accumulation as a lower wall-clock rate. So that spread discriminates nothing, and I should not have leaned on it. The small-corpus test in a quiet window remains the clean discriminator, and the three indexes-off question stands alongside it.

No pressure on timing for any of this. You have already given us more than most reports ever do, and the next move is ours: ship the resolution fix so the measurement is cheap for you.

The aft index resolution fix is on main.

What it was: doctor --fix installs to <cache>/v<version>/aft, and the resolver only searched that directory when it was handed an explicit version — which aft index never does. So it checked the platform npm package, then PATH, then ~/.cargo/bin, and in a container with none of those it reported "requires a native AFT binary" while pointing you at the command you had already run.

The same blindness affected aft --version and aft doctor --profile; on my own machine --version reported "aft binary not installed" with three versions sitting in the cache. Fixed at the shared resolver, newest-by-semver so v0.10.0 beats v0.9.0.

The not-found message now lists every location it searched and why each was empty, rather than naming a remedy you may have already tried.

This ships in the next release. If you would rather not wait, AFT_BINARY_PATH=<path to the cached binary> works today on the version you have — the cache layout is ~/.cache/aft/bin/v<version>/aft, or %LOCALAPPDATA%\cortexkit\aft\... on Windows.

No obligation to run anything. The small-corpus test in a quiet window is the clean discriminator whenever you have the time for it, and the three-indexes-off run is the other one — but you have already given this more than most reports get, and we have a Linux reproduction path of our own now.

bpmn-aft-12447.log
bpmn-rss-track2.log
bpmn-smaps-1.txt
bpmn-smaps-2.txt

Answers to your three questions, from a fresh instrumented leaking run (v0.56.2, real binary, remote openai_compatible backend, the 44,200-file corpus, Linux container, no quota):

2. smaps_rollup — it is the heap. Two snapshots four minutes apart during the balloon:

	15:15Z	15:19Z
Rss	17.2 GB	21.7 GB
Pss_Anon	17.97 GB	22.77 GB
Pss_File (mapped artifacts)	41.9 MB	7.7 MB
Private_Dirty	17.97 GB	22.77 GB
Threads	42	—

Virtually the entire balloon is anonymous, private, dirty — mapped files are a rounding error. This is allocation, not artifact mapping or page-cache accounting.

3. Complete daemon log from the leaking run. One finding first: exactly one build_started in the whole log — no concurrent semantic builds. The run progressed normally (no errors, no skips logged) and was building the whole time: it reached batch 125/5872 (8,000 chunks) in ~9.5 minutes while RSS went 0.6 → 22.8 GB. That works out to ~178 MB of heap per batch against your harness's 0.19–0.23 MB/batch — three orders of magnitude. The 30-second-interval RSS trajectory and the two smaps snapshots are attached alongside the log; the run ended in a manual SIGTERM (code=143) to protect the host rather than the watchdog, so the log shows no kill event.

The synthetic-corpus caveat you named — content-shaped rather than count-shaped — now looks like the live hypothesis: our corpus is the one thing your scale run did not model (real Cyrillic .properties, dense XML, base64 literals — the same content classes from #318). If the accumulator keys off something in the row content or the recovery path's interaction with it, a uniform synthetic corpus would be flat exactly as you measured.

1. Indexes-off run is queued next on our side; I'll report either way.

I read all four attachments rather than only the summary, and they move this a long way. Three things your data settles that our own runs could not.

Collect is not the balloon. Your log shows semantic collect: 375756 chunks from 37212 files in 10945 ms at 15:10:28, eleven seconds after build_started and about four and a half minutes before your RSS track begins. Whatever accumulates does so during embedding, not during collection — which rules out the first thing I would have chased.

The rate is higher than your average and not constant. Between 15:15:09 (15.91 GB) and 15:16:09 (22.15 GB) you cross roughly twenty batches, which is about 310 MB per 64-chunk batch rather than the 178 MB the whole run averages. That matters because a batch's own data is bounded: the embedded text per chunk is capped at ~1600 characters and the display snippet at five lines or 300 characters, so 64 chunks is on the order of 100 KB in and a fraction of a megabyte of vectors back. Whatever allocates 310 MB is not the batch's payload.

It plateaus, and that is a clue rather than a detail. Your track reaches ~22 GB and then oscillates and falls — 22.49, 22.52, 22.01, 23.33, 22.47, 21.53, 20.86. A ceiling with oscillation looks like something proportional to the corpus being accumulated and levelling off, not an unbounded per-batch leak. Any explanation that predicts indefinite growth is contradicted by your own data. Worth noting too that between your two smaps snapshots Swap went 0 to 11.5 GB, so committed anonymous memory is ~34 GB and the machine was paging hard by the end.

On your content hypothesis, I think you are right and I can sharpen why. Our chunk text is capped in characters, not tokens. Sixteen hundred characters of Cyrillic or base64 tokenizes to far more than sixteen hundred characters of English — plausibly past a server-side 512-token limit. That is exactly the #318 class you pointed at, and #318's remedy is recursive batch bisection with chunk shrinking. A corpus where most batches trip that recovery path is a shape we have never run, and a recovery that allocates per bisection level would look precisely like this: large per-batch growth, content-dependent, levelling once the pathological set is exhausted.

That is now the first arm being measured rather than one of several, with the recovery path instrumented directly rather than inferred from total RSS. If it comes back flat the hypothesis is dead and I will say so.

No need to run the indexes-off test on our account any more — your instrumented run already answered more than it would have. Run it if it is cheap for you, but you have given this more than enough.

My sharpened hypothesis was wrong, and the measurement says so clearly. Reporting it before you spend any more of your time.

Nine content arms — Cyrillic properties, dense XML, long base64 lines, a rotating mix, and controls — all flat, in a 0.02 to 0.53 MB/batch band against your 178 MB/batch average. Content is not the variable.

Two specific things I got wrong, both worth you knowing because they also change what your corpus can do:

The oversized-chunk recovery path is not an accumulator — it is the opposite. Two arms drove it on essentially every batch (51,187 and 76,287 context-overflow rejections, requests per batch rising from 1 to 188 and 249) and produced the lowest slopes of any arm in their family. An allocator-counted test puts it exactly: 3,416 bytes retained per chunk with the limit enforced against 3,886 without, so 0.88x. A shrunk row stores less than the row it replaced. Recovery makes a build smaller.

Multi-byte content does not lengthen a row. I assumed the text cap was in characters; the body cap is actually compared and sliced against byte length. So Cyrillic yields the same byte length with fewer characters, and the only divergence is token count — 1.48x, 466 tokens against 314 for an identical 1,097-byte row. That does not cross a 512-token limit, so a Cyrillic body cannot produce the overflow I predicted at your settings.

And one finding that may matter more than either: .properties, .xml and .bpmn are not in the semantic indexer's extension list at all. The file types you named never become chunks and never reach an embed batch. Whatever your 375,756 chunks are, they are coming from the other files in that tree — which also means a corpus built only from the classes you described would not reproduce it even in principle.

So the search moves back to what is still unmodelled: your actual chunk shapes, rather than the content classes they come from. If you are willing, the single most useful artifact now would be the chunk-size distribution from a leaking run — or failing that, the language breakdown of the 37,212 files that did produce chunks. That tells us what to generate instead of guessing again.

Caveat on our side, stated plainly: our token model is a two-class approximation, charging ASCII at 3.5 characters per token and non-ASCII at about one. A real tokenizer splits high-entropy ASCII such as base64 far more finely, so a real corpus could trip the overflow path where our arms did not. The recovery arms cover the consequence by forcing that path directly, but the claim that Cyrillic cannot overflow rests on that model and should be read as such.

0.57.0 ships shortly with the fixes that are ready. This issue stays open and the investigation continues.

1. Indexes-off run — done, and it stops the leak. Same daemon (v0.56.2), same corpus, same remote backend, same scarred stores — with search_index: false, callgraph_store: false, views: false (validated against assets/aft.schema.json; callgraph_store is a real key, just undocumented in config.md):

	all planes on (previous run)	three planes off (this run)
min 9	dead at 22.8 GB, batch 125	0.9 GB, building
min 67	—	1.02 GB flat (+0.15 MB/min drift), batch 1985/5872, 127k chunks embedded

Every balloon onset we have observed falls in a 3–60 minute window; this run passed all of them and embedded 16× the batch count of the dead run at 1/20th the memory. Per your framing: the growth is in a plane running beside the build, not in the semantic embed path itself. views was already false in the leaking run, so the remaining candidates are the trigram index and the callgraph store — the latter being the one whose persistent state our crash history has been scarring (the SIGKILL/stale-state report I'll file separately covers exactly that store).

One side observation from the run: with callgraph_store: false the tier2 dead-code aggregation attempts still fire and fail (tier2_dead_code_aggregate_did_not_complete) — the flag does not gate that plane, if that matters to your search.

This is the differential that turns the investigation around, and it is one we could not have run — it needed your corpus and your scarred stores. Thank you.

Sixteen times the batches at a twentieth of the memory, through every onset window you have observed, is not ambiguous. The embed path is exonerated: the accumulator is in a plane running beside the build. With views already false during the leaking run, that leaves the trigram index and the callgraph store.

Two things on our side line up with that, one of them closely enough to be worth your attention.

A Tier-2 code-health scan has been starting while the same project's callgraph cold build is still filling the store it reads. I landed a fix for exactly that overlap earlier today. On a mid-sized benchmark repository it cost about 90 MB, with a call-graph build of 97 seconds against a 90-second timer — the scan only overlapped the last few seconds. Your corpus is 37,212 files producing 375,756 chunks, where the cold build runs very much longer, so the same overlap has far longer to accumulate. That is now the first arm being measured, with and without the fix, at your shape.

Your side observation is a real defect and I am glad you included it. Reading the source: the readiness predicate does honour callgraph_store: false and refuses the snapshot, which is why you see the aggregation fail rather than produce wrong results. But something upstream is still dispatching the job — a disabled plane should not be scheduling work at all. That is being located now. It is probably not the leak, but it is wrong on its own terms.

Four arms are running, each differing from your control by exactly one flag, so we can name the plane rather than the pair. If neither single plane balloons while both together do, that is an interaction and I will report it as one rather than forcing a culprit.

Please do file the SIGKILL/stale-state report separately. If the callgraph store's persistent state is being scarred by crash history, that is worth its own thread regardless of how this one resolves — and it may well be the reason your instance behaves differently from every clean-store run we have managed.

0.57.0 is being cut now. It does not contain a fix for this, and I am not claiming one; it carries the Tier-2 overlap fix and a good deal else, and this issue stays open.

Correction to my previous comment, before you act on it.

I called your side observation "a real defect" and said a disabled plane should not be scheduling work at all. I had not read the relevant code when I wrote that — I was relaying a conclusion rather than a reading, and it is at least partly wrong.

The automatic Tier-2 refresh is gated. automatic_tier2_refresh_categories (context.rs:6647-6662) reads config.callgraph_store and filters dead-code out when it is false, with a comment noting that zero contributions would force a full rescan and discard, so the work would be pure waste. I have now read those lines myself.

So something else reaches the dead-code aggregation in your run, and it may well be legitimate — an explicit inspect request is a user asking for that scan, and silently refusing it would be its own kind of dishonesty. The honest outcome here may be "not a defect, but the log line is misleading", which is a fine answer and not the one I gave you.

What stands: you are seeing tier2_dead_code_aggregate_did_not_complete with the flag off, that is worth explaining, and it is being tracked down. What does not stand: my characterisation of it as a defect. Apologies — you have been careful with your evidence throughout this thread and you deserve the same back.

Nothing else in that comment changes, and the plane-isolation arms are unaffected.

We have it. The plane is the trigram search index, not the callgraph store, and not the embed loop at all.

Six arms on a 37,212-file corpus — your file count — with one flag differing at a time:

arm	planes	peak RSS
B1	search_index only	5,831 MB
B2	search_index only	5,840 MB
C1	callgraph_store only	837 MB
C2	callgraph_store only	841 MB
B-build-only	search_index, no embedding at all	5,832 MB
B-quarter	search_index, 9,303 files	1,513 MB

Three things fall out of that.

It reproduces without embedding. The build-only arm never runs a single embed request and still reaches 5,832 MB. So the embed loop was never the mechanism — it was simply the thing running alongside the real one, which is why disabling the three planes stopped your balloon.

The callgraph store is exonerated. Both C arms sit under 850 MB, flat. I had steered my own investigation toward a Tier-2 and callgraph overlap earlier in this thread; that was wrong, and the C arms are what disproved it.

It scales with corpus. Quarter the corpus, quarter the memory: 9,303 files gives 1,513 MB against 37,212 files giving 5,831 MB — four times the input, 3.85 times the peak. That is the answer to why you see this and our earlier synthetic runs did not.

And the shape matches your trace: the smaps rollup from the quarter arm is 243 MB of 243 MB anonymous private dirty, with mapped files a rounding error — the same composition as your 17.97 GB Pss_Anon reading. The plateau also releases in our arms exactly as it did in yours, which is why your RSS oscillated back down rather than climbing forever.

One honest gap. At your file count we reach 5.8 GB where you reached 22.8 GB. Our corpus is generated, so its files are almost certainly smaller and produce fewer chunks than your 375,756. The mechanism is the same; the constant is not. So please read the ratio as confirmed and the absolute number as ours rather than yours.

Still to do: name the exact allocation inside the search index build. The shape — held for tens of seconds during construction, released afterwards, proportional to corpus — points at cold trigram index construction, and I want the line cited rather than inferred before I claim it.

A workaround you can use tonight, if it helps: search_index: false costs you trigram search on that project and leaves semantic and callgraph intact. Your own three-plane run already showed that path is stable for over an hour.

Thank you for the differential. We would not have found this without it.

Root cause found, and it is not where either of us was looking. It is not the trigram build — it is a symbol prewarm that runs after the trigram index publishes, on the same corpus.

prewarm_symbol_cache_from_search_files creates one parser before its loop and reuses it for every file in the corpus:

let mut parser = FileParser::with_symbol_cache_generation(...);
for (path, modified) in &symbol_files {
    match parser.extract_symbols_with_cache_status(path) { ... }

And FileParser holds cache: HashMap<PathBuf, CachedTree> — one retained tree-sitter syntax tree per path, with no eviction. So a whole-corpus prewarm accumulates a parse tree for every file and holds them all until the loop ends. On your 37,212 files that is 37,212 resident syntax trees.

That explains every part of the shape you measured:

Anonymous private dirty, because these are heap-allocated trees rather than mapped artifacts — matching your 17.97 GB Pss_Anon reading.
Proportional to corpus, because it is one tree per file, sized by source bytes and syntax complexity. Our quarter-corpus arm landed at 1,513 MB against 5,831 MB for the full one.
A plateau that releases in one step, because the parser is dropped when the prewarm function returns. Nothing is reclaimed gradually; it all goes at once. Your RSS oscillating back down was this.
Reproducing with zero embed requests, because the prewarm is triggered by the search index's file list, not by embedding.
And why it correlates with the embed build at all: the prewarm is spawned on its own thread as configure completes, so it runs concurrently with embedding and looked like part of it.

This is accidental retention rather than a design cost. The prewarm needs each tree only until that file's symbols are extracted; nothing requires keeping the previous 37,211. The documented 128 MiB SPIMI budget is real and was never the constraint here, because it governs the posting block in a different phase.

Your workaround stands and is now explained: search_index: false skips the file list that triggers this prewarm.

One honesty note on the attribution. We have the source lifetime, the event ordering, the near-exact 4x scaling and the release point, but not a heap allocation trace. I am confident, and I would rather tell you the basis than imply more certainty than we have.

Fix next, and it is a small one — evict each tree once its symbols are extracted. I will link it here when it lands, and it will name you in the release notes.

Fixed on main as of 7b8fab3ba.

The change: the symbol prewarm now evicts each parse tree once that file's symbols are extracted, instead of holding all of them until the loop ends.

Measured on a generated corpus at your file count, prewarm peak RSS:

corpus	before	after
37,212 files	5,246.9 MB	320.0 MB
9,303 files	1,349.1 MB	115.9 MB

Through the full harness, including the semantic state that is legitimately corpus-sized, 37,212 files went 6,032.4 MB to 1,252.5 MB. Prewarm wall time is unchanged — 12.817s against 12.814s across alternating three-run medians — so nothing was traded for the memory.

Two details on how it was done, since you have been careful with yours. The eviction is a narrow tree-only method; invalidate_symbols now delegates to it with unchanged semantics, so every other caller keeps normal tree-cache reuse — the cache is genuinely useful where a file is parsed repeatedly, and only a whole-corpus single-visit pass was the outlier. And the regression test checks the parser after every file rather than at the end, then compares every retained symbol vector against independent extraction, so it proves both that the trees are gone and that your symbol data is byte-for-byte unaffected.

It will ship in the next release; I will comment here when that is published and close this then rather than now, so the issue stays open until you can actually install the fix.

On the scale question, honestly: we reached 5.8 GB where you reached 22.8 GB at the same file count, because our generated files are smaller and produce fewer chunks than your 375,756. The eliminated component is the per-file retained tree, which is the part that scaled. Whatever remains on your corpus should now be proportionate to the index data we intend to keep — if it is not, that is a second finding and I want to hear it.

Thank you for the three-plane differential. Three of my own hypotheses died to your data, and the one that survived was only reachable because you ran the experiment we could not.

A second finding on v0.57.0, distinct from the prewarm leak — and a question about where it belongs.

Context. After the indexes-off differential we left search_index: false (your workaround) and restored callgraph_store: true. The daemon auto-updated to v0.57.0 — which I understand predates the prewarm fix, so that fix was not in play; the prewarm path was disabled by the workaround regardless.

What happened. On a fresh session with the semantic build running, RSS reached 15.6 GB by minute ~19 (clean SIGTERM to protect the host). The daemon's own log names the mechanism — a Tier-2 snapshot materialized from the persistent store at minute ~13:

perf tier2_callgraph_snapshot: source=callgraph_store
  files=37358 exports=331300 edges=2541263 entry_points=44 ms=50465

The store this projects from is 3.4 GB and crash-scarred: it survived a day of five watchdog SIGKILLs (Sept 17, 09:38–11:26), each kill leaving state the next session had to re-materialize. A stable twin sandbox's store measures 2.1 GB. Our Sept 16 record describes the full loop: SIGKILL → scarred state → next session balloons assembling this snapshot → OOM → SIGKILL; parking the store dirs buys exactly one stable build cycle before the refill re-arms it. With the prewarm leak now understood as the thing that drove most of those OOMs, what remains is the snapshot materialization itself — and today shows it reaching 15.6 GB alone, with the prewarm disabled.

I don't read this as contradicting your C arms: those ran fresh synthetic stores under 850 MB, and the mechanism here is the scarred 3.4 GB one. Your exoneration stands for the code path on clean state; the open question is whether survival across SIGKILLs — and bounded materialization of a large store — is a property the daemon should have.

One operational observation that may matter for the fix design: I tried parking the store directories while a (TERMed) daemon was shutting down, and the shutdown's callgraph flush re-wrote the full store before exit. A recovery or compaction step can't assume the dying process leaves the store untouched.

The question. Earlier in this thread you asked us to file the SIGKILL/stale-state behaviour as a separate issue. This snapshot balloon is that mechanism, now reproduced on v0.57.0 with the prewarm disabled. Should we file it separately (we have the full record: the five kill timestamps, store sizes, RSS trajectories, the reset-buys-one-cycle pattern), or would you rather track it here alongside the prewarm fix — for instance if the planned release changes how Tier-2 snapshots are assembled? Config on our side is back to callgraph_store: false, which is stable.

File it separately, please. It is a different mechanism from the prewarm leak and it deserves its own thread, its own reproduction and its own fix.

The two are unrelated in the code. The prewarm leak was in symbol extraction: prewarm_symbol_cache_from_search_files held one tree-sitter syntax tree per file for the whole corpus, so it scaled with file count and released at the end of the pass. What you are seeing now is the Tier-2 dead-code projection materializing a snapshot out of the persistent callgraph store — a different subsystem, a different allocation, and a fix that will not be the same shape. Tracking it here would bury it under a closed finding.

Your numbers make the case on their own. files=37358 exports=331300 edges=2541263 ... ms=50465 reaching 15.6 GB, with the prewarm path disabled by the search_index: false workaround, is a clean second observation. And you are right that it does not contradict the C arms: those ran fresh stores under 850 MB, so they exonerate the path on clean state and say nothing about a 3.4 GB scarred one.

Two things in your report that I want carried into the new issue verbatim, because they are the parts I would not have known to ask for.

The first is the loop. SIGKILL leaves state the next session must re-materialize, the re-materialization balloons, and that is what earns the next SIGKILL. Parking the store buys exactly one clean cycle before the refill re-arms it. That is a self-sustaining failure, not a series of unlucky builds, and it means a fix that only bounds memory without addressing what a killed process leaves behind will look like it worked for one cycle.

The second is the operational note about the shutdown flush rewriting the full store on the way out. That constrains the fix directly: any recovery or compaction step cannot assume a dying daemon leaves the store untouched, and a naive "repair on next open" would race exactly that write. I would have got that wrong.

On the design question you are really asking — whether bounded materialization and survival across SIGKILLs are properties the daemon should have — yes to both, and neither is true enough today. Tonight I landed work that narrows the surface without closing it: Tier-2 now waits for a callgraph cold build on the same root instead of running concurrently with it, and the dead-code projection got an incremental path so it splices a journal rather than re-projecting the whole store whenever it can. Neither bounds peak memory for a full materialization of a store your size, which is what your trace shows.

So: open it, link this thread, and put the five kill timestamps, the store sizes and the RSS trajectories in it. I would rather have your record than reconstruct a worse one.

Separately, the prewarm fix is on main and will ship in the next release; I will comment when it does. Keeping callgraph_store: false is the right call meanwhile, and I am sorry it is the second workaround you are carrying for us.

v0.57.1 fixes the first of the two mechanisms in this thread. Leaving this open until you can confirm at your scale.

What was fixed. prewarm_symbol_cache_from_search_files created one parser and reused it across every file. The parser caches a tree-sitter syntax tree per path with no eviction, so the prewarm pass held one parsed tree per file on the heap until it finished. That matches every property you measured: private dirty memory, linear growth with file count, release at the end of the pass, and reproduction with zero embedding requests.

The fix evicts each tree immediately after its symbols are read, rather than changing the cache for callers that legitimately re-parse the same file. On a corpus your size (37,212 files), isolated prewarm peak went from 5,246.9 MB to 320.0 MB, with wall time unchanged (12.817s vs 12.814s across alternating medians).

What is not fixed. The second mechanism you found with prewarm disabled — Tier-2 dead-code projection reaching 15.6 GB — is tracked in #330 and is untouched by this release. An inventory of that path is complete and confirms the plateau is not confined to projection: rollup builds an owned edge corpus and two reachability relations after projection completes, so bounding the projection alone would not bound the peak.

Two things that would help when you upgrade: whether embedding-time RSS is flat for you now with prewarm enabled, and whether the remaining growth matches the #330 shape (dead-code specifically, rather than embedding).

Your plane-isolation runs are what made the first mechanism findable — holding at 1.02 GB with the three index planes disabled ruled out the embedding pipeline in one measurement.

Confirmed at our scale on v0.57.1 — both of the measurements you asked for.

Setup: real corpus (~37,416 indexed files), remote openai_compatible backend, search_index: true (prewarm enabled), callgraph_store: false (the #330 workaround), views: false, Linux container, no limits.

Cold prewarm. We parked the on-disk symbol cache to force the hard case:

loaded symbol cache from disk: 0 files
pre-warmed symbol cache: 37416 new, 0 cached, 37416 files total

All 37,416 files parsed fresh. RSS through the pass peaked at 0.74 GB and settled the moment it completed — against the pre-fix behaviour on the same corpus (your isolated 5.2 GB harness number; our combined runs died at 15.6–22.8 GB).

(a) Embedding-time RSS with prewarm enabled: flat. A warm-cache run held 0.83 GB for its duration (we terminated it deliberately to run the cold case), and the cold run has now been embedding for 60 minutes at 0.73 GB with +0.3 MB/min drift, at batch 1175/5872 (~75k chunks). The pre-fix runs on this corpus were dead by batch 125 at 22.8 GB. Every balloon onset we ever observed fell in a 3–60 minute window; this run has passed all of them at 1/30th of the memory.

(b) Remaining growth: none in the embed phase. The build is still running to completion (~4 h at our shared-backend rate) and we'll report if that changes, but there is nothing growth-shaped in the embed lane. The #330 mechanism stays excluded by callgraph_store: false, as intended.

Attachments: warm and cold RSS trajectories (30 s cadence) and the cold-run daemon log.
That's the full loop on #327 — filed, root-caused, fixed, and validated from our side. One workaround remains (#330, design phase). The build finishing in a few hours just leaves bpmn with a complete index.

bpmn-v571-cold-daemon.log
bpmn-v571-cold-rss.log
bpmn-v571-warm-evidence.log

Thank you for measuring it end to end, and on the hard case: a cold prewarm peaking at 0.74 GB across 37,416 files, then flat through embedding, where the same corpus used to die at 22.8 GB. That confirms the fix at your scale, which our own benchmarks couldn't, so I'm closing this one.

On #330: the fix for the largest part of that memory is on main. It measured at your file count at 7.03 GB peak RSS before, 2.2 to 2.5 GB after (details on #330). It will be in the next release. When that's out, a run with callgraph_store: true on your store is the test that tells us whether you can drop the workaround.

Navigation Menu

Repository navigation

Embed builds leak daemon RSS on ≥0.56.1 — remote backends always, local fastembed on 0.56.2 #327

Short description

What happened?

Diagnostics

Plugin version

AFT binary version

Platform

Log output (optional)

Two answers

1. Container limits / cgroup reader

2. Small-corpus reproduction

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Embed builds leak daemon RSS on ≥0.56.1 — remote backends always, local fastembed on 0.56.2 #327

Description

Short description

What happened?

Diagnostics

Plugin version

AFT binary version

Platform

Log output (optional)

Activity

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

OlegAndreych commented on Sep 21, 2026

Two answers

1. Container limits / cgroup reader

2. Small-corpus reproduction

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

OlegAndreych commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

OlegAndreych commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 21, 2026

aft-alfonso commented on Sep 22, 2026

aft-alfonso commented on Sep 22, 2026

aft-alfonso commented on Sep 22, 2026

OlegAndreych commented on Sep 22, 2026

aft-alfonso commented on Sep 22, 2026

aft-alfonso commented on Sep 22, 2026

OlegAndreych commented on Sep 23, 2026

aft-alfonso commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions