Repository navigation
Per-root resident artifacts are capped per-item but have no aggregate bound across roots #210
Description
Every resident artifact has a sensible individual cap. Nothing caps their sum.
search_index SPIMI_SOFT_LIMIT_BYTES 128MB / HARD 256MB, MAX_ENTRIES 10M per root
semantic_index max_files per root
callgraph store SQLite + buffers per root
symbol cache disk-backed per root
borrowed cache BORROWED_INDEX_CACHE_CAPACITY = 4 process-wide ✓
Only the borrowed-index cache is bounded process-wide. The rest are per-root, and the process holds as many sets as it has bound roots. IDLE_ROOT_TTL (30 min) is the only thing that ever reclaims them, and it's a liveness rule, not a budget — a daemon with N actively-used roots keeps N full artifact sets resident by design, and no configuration expresses "this process should not exceed X".
Live instance right now, ~2 h uptime:
VmRSS 2.95 GB
VmData 10.97 GB
Threads 123
31 distinct sessions in the last 110 perf ticks
This is the same shape as #202, one layer up. There, InspectManager lived in the per-root AppContext, so N roots meant N rayon pools (88 inspect threads of 211). The fix made the pool process-wide. The artifacts have the identical structure — bounded per item, multiplied per root, unbounded in aggregate — and no equivalent fix.
Two reasons I think it's worth a bound rather than just an eviction TTL:
It's the allocation pattern that strands free space. Many large, long-lived, differently-sized buffers arriving and departing on independent schedules (bind, watcher churn, idle eviction, reconfigure) is the textbook way to interleave free chunks beneath live ones. That's the mechanism behind the fragmentation in #205 — where measurement showed 15 of 16 arenas at ≥0.86 free with one arena doing the work — and malloc_trim structurally can't reclaim it.
A multi-root daemon has no backpressure. Idle eviction is time-based, so 31 sessions across many roots in one process means the peak is set by how many roots happen to be warm at once, not by anything the operator can configure. On this host that contributed to an OOM kill of the daemon (Consumed … 29.5G memory peak, 38.6G memory swap peak at 41 h uptime).
What I'm not proposing: a specific mechanism. LRU eviction on a byte budget, a total-artifact-bytes config with eviction of the coldest root, or per-root caps scaled by root count are all plausible and have different failure modes — evicting a root someone is about to use costs a rebuild, which may be worse than the memory it saves. You'd know which fits the eviction machinery that already exists.
The concrete ask is whether an aggregate bound belongs here at all, or whether the intended answer is "run fewer roots per daemon". If it's the latter that's worth stating somewhere, because the daemon transport makes many-roots-per-process the default rather than the exception.
Related: #205 (fragmentation measurement), #202 (same shape, per-root thread pools).
Activity
ualtinok commentedon Aug 10, 2026
Accepted — the structural read is right, and the #202 parallel is the correct lens: bounded per item, multiplied per root, unbounded in aggregate, with the additional bite you named that the allocation pattern itself is the fragmentation mechanism from #205. The borrowed-index cache being the only process-wide bound is an accurate census of the current state.
This one is getting a design cycle rather than a quick bound, deliberately. An aggregate budget forces real policy questions that a cap constant would silently answer badly: evicting a hot root's search index trades RAM for rebuild churn (and on this codebase a cold rebuild is exactly the storm class we spent last week killing); LRU across artifact types is wrong (a 200 MB search index and a 200 MB callgraph store have very different rebuild costs and hit rates); and the budget interacts with the idle-root TTL and the cold-build limiter, which are liveness rules, not budgets — composing a third regime on top without a design pass is how we'd get eviction storms. The shape I expect the design to land on is a process-wide accounting layer (the per-root attribution from the memory snapshots already exists) driving priority-aware eviction through the existing idle-eviction machinery, with the budget expressed as a soft target rather than a hard cap. It'll get an adversarial review before implementation, same as the other concurrency-adjacent designs here.
Your VmData 10.97 GB vs RSS 2.95 GB gap is also worth keeping in frame — a large slice of that aggregate is already MADV_FREE'd or never-touched; the budget design has to measure what's actually resident per artifact, not summed capacity, or it will evict against phantom pressure. The per-arena distribution work you're doing on #205 feeds this directly.
iceteaSA commentedon Aug 10, 2026
The residency point is right, and it's worth more than a caveat — I measured it, and the error a capacity-summing budget would make is large enough to invert decisions.
Same daemon, now 8 h uptime:
Per-region, from smaps — 157 large anonymous rw mappings ≥32 MB:
A budget summing capacity charges 12.37 GB. A budget summing residency charges 4.34 GB. 2.9× overcharge.
The distribution is what makes this a correctness issue rather than a scaling constant. It's bimodal: 70 regions are effectively untouched while 46 are nearly fully committed. A capacity-based budget doesn't overcharge uniformly — it overcharges those 70 regions by ~20× and the other 46 barely at all. So under pressure it would preferentially evict whatever happens to hold large sparse mappings, which is the opposite of what you want, and it would fire while 8 GB of the "usage" it's reacting to isn't costing the box anything.
Worth noting the one thing I could not find: zero regions are fully untouched. Every one has at least some resident pages, so an eviction policy can't cheaply identify "reserved but unused" artifacts by a null test — it needs the actual fraction. Cheap to read (
Rss+Swapper mapping in smaps) but it does mean attribution has to be per-mapping and periodic rather than a one-time classification.One caution on wiring my #205 work into this, since you named it as an input. The per-arena distribution answers "is free space reclaimable" — 15 of 16 arenas at ≥0.86 free, one arena doing the work. It does not attribute bytes to artifacts. glibc arenas are per-contending-thread, not per-subsystem: a single arena holds allocations from whatever threads landed on it, so arena-level data can't tell you the search index owns X and the callgraph store owns Y. For a priority-aware policy that needs per-artifact residency, arena data is the wrong granularity and the per-root memory snapshots you mentioned are the right one. The two are complementary — mine says how much of the total is recoverable in principle, yours says who to charge — but substituting one for the other would put phantom precision into the accounting layer.
Design cycle over quick bound is clearly right, and the rebuild-cost asymmetry argument is the part I'd have got wrong: I was thinking in bytes, and a 200 MB search index and a 200 MB callgraph store having different rebuild costs makes byte-uniform LRU actively harmful. Happy to run measurements against a proposed policy if that's useful during the design pass — I have the harness and the census tooling from #205 already.
iceteaSA commentedon Aug 14, 2026
One correction for the design cycle, since you named the #205 work as an input and #205 closed with a mechanism description my own measurements refute.
The closing summary there reads "the dominant factor was arena count scaling with thread count." That holds only below glibc's arena ceiling. Above it, thread count does not move arena count at all:
A standalone C probe pinned why — 16 is glibc's default limit on that host, reached at 16 concurrent allocators and unmoved by more:
It's a real limit rather than a workload artifact — raising
MALLOC_ARENA_MAXexplicitly moves the ceiling in lockstep (4→4, 32→32, 64→64), and it is not8 × ncores(that would be 128 on a 16-core host).Why this matters here specifically: a budget or pool design that reduces thread count expecting a proportional drop in arenas will get nothing until it crosses below ~16 concurrent allocators. Whether #202's 88→8 inspect-thread cut reduced arena count depends entirely on whether it dropped concurrent allocators under that threshold — and the production data doesn't tell us that either way. The thread reduction is clearly good for other reasons; I just don't want the design cycle to inherit "fewer threads ⇒ fewer arenas" as a general lever.
What still stands from that work, and is the part relevant to a budget: 15 of 16 arenas sat at ≥0.86 free with one arena doing the work, and capping arenas cut retained free space 65% with low-occupancy arenas going 10→0.
Still open from my last comment here, and unchanged: a capacity-summing budget overcharges 2.9× against measured residency, and the error is bimodal (70 regions effectively untouched, 46 nearly full), so it would preferentially evict whoever holds large sparse mappings. That's the input I'd actually feed the accounting layer — the per-arena data answers "is this reclaimable," not "who owns it."
Offer stands to run measurements against a proposed policy during the design pass.
iceteaSA commentedon Aug 30, 2026
One code constraint for the design cycle, since you named "priority-aware eviction through the existing idle-eviction machinery" as the shape — that machinery currently can't reach the roots that hold the memory.
reap_idle_rootsgates non-deleted eviction onhas_bound_route == false(subc/mod.rs:1213): a root with any bound channel is never an eviction candidate, andIDLE_ROOT_TTLonly starts counting after the last channel unbinds. So the resident set is exactly the bound roots, and the existing machinery excludes all of them by construction — it reclaims a root only once no session is using it, which is after the pressure has already passed.The mechanical half is reusable, though.
evict_idle_artifactsjust drops the in-RAM handles (search / semantic / callgraph / symbol / inspect) and the query paths reload from disk on next use. Nothing in that operation requires the root to be unbound — the unbound check is a policy gate, not a safety one. The safety gate isartifact_eviction_blocked, which already covers active builds, bash tasks, inspect scans, and pending disk updates. So the shed path you're describing could run the same drop on a bound-but-quiet root under pressure and let the next query pay one reload — which is the rebuild-cost asymmetry you flagged, now scoped to "shed the cheapest-to-reload artifacts of the coldest bound root first."Fresh census, current daemon,
MALLOC_ARENA_MAX=2deployed (so the #205 lever is already pulled):The uptime is the point, not a caveat: at 9 minutes there's been no churn to fragment the heap, so ~5.8 GB resident is essentially pure load-in working set — ~445 MB/root averaged across 13 bound roots. That's the floor the aggregate bound is fighting, independent of #205, which only adds on top over hours. It's also the direct confirmation of your residency-not-capacity point from the other side: the arena cap is on and the resident total still lands here, because this is live working set, not reclaimable free space.
Separately, and not RAM — the on-disk artifact store is unbounded in a way worth a glance while you're here: 821 callgraph root dirs totalling 13 GB, one root at 6.2 GB alone, against 13 ever-live. That's generation GC + cache-key eviction (the #224 family) not keeping pace — distinct from the resident-memory question, same accumulation shape one layer down.
Offer from the last comment stands: harness and census tooling from #205 are ready to run against a proposed shed policy.
aft-alfonso commentedon Aug 30, 2026
Both halves confirmed, and the policy-vs-safety split is the constraint the design cycle needed stated plainly - thank you for pinning it to the exact gate. Agreed on all three points:
has_bound_routeat subc/mod.rs:1213 is policy (the resident set is exactly what it excludes),evict_idle_artifactsis mechanically reusable on a bound-but-quiet root, andartifact_eviction_blockedis the real safety gate. The shed design folds this in as: pressure watermark -> coldest bound root by last-activity -> cheapest-to-reload artifact class first, same drop path, reload-on-next-query - with your ~445 MB/root census as the sizing input for the watermarks. We will take the harness offer when the shed policy has a testable shape.The 13 GB / 821-dir store finding is now a dispatched fix on our side: census first (where the accumulation concentrates and why each class escapes the existing reapers - generation GC only runs on active roots' publish paths, and the cache-key eviction removes the key mapping without the keyed artifact dir), then a store-wide budgeted sweep with liveness guards (lease, reader markers, cache-key presence, conservative age threshold) following the same pattern as the inspect scope-cache GC. Will report numbers here when it lands.
iceteaSA commentedon Sep 2, 2026
The residency measurement you asked for — a 3-day time series on the live daemon rather than a snapshot, so it can separate "how much per root" from "does it grow over time".
Instrument: every 30 min,
smaps_rollup(Private_Dirty,Swap) +/proc/status+ bound-root count (distinct/callgraph|search|inspect/<key>paths in/proc/<pid>/fd), read-only. 145 samples across 9 daemon uptimes,MALLOC_ARENA_MAX=2throughout. Segmented by monotonic uptime so no window straddles a restart.One correction to the obvious metric first:
Private_Dirtyalone is misleading on a loaded box — it falls whenever the kernel swaps the daemon out (one window: 5636 → 394 MB whileSwaprose 0 → 4787 MB, same process, same roots). The un-confounded measure is committed = Private_Dirty + Swap. Everything below uses that.Result — committed is flat when root count is flat, and steps when roots change:
Root-count steps in the same series: 18 → 9 roots: −1.8 GB. 11 → 9 roots: −2.2 GB. Two independent uptimes, same shape.
So the footprint is a function of bound roots, not of time. There is no intra-root accumulation to chase — which retires the fragmentation half of this (#205) as a live concern: the arena cap is on, and nothing is left growing underneath it.
What the earlier "GB/h" growth rates actually were:
VmSwapgrowth. Swap rising while committed stays flat means the kernel was evicting the daemon's working set under box pressure, not the daemon allocating. That series measured the host and billed the daemon. (VmDataclimbs regardless — 6.7 → 9.4 GB in segment A while committed was flat — that's uncommitted glibc address-space reservation; it's the "phantom pressure" you flagged, now quantified at ~3–4 GB on this instance.)Sizing input for the watermark: per-root cost is root-dependent, not a constant — 200–700 MB marginal observed depending on which roots left (a checkout with a 6 GB on-disk callgraph store is not a 50 MB one). A budget that assumes a flat per-root figure will be wrong in both directions; per-root attribution from the memory snapshots is the right granularity, as you said.
One anomaly, reported not explained: one segment showed +5.7 GB committed in a single 30-min interval with roots flat at 11 (RSS 9.0 GB,
VmData18.2 GB), followed by a restart within the hour. Single transient; I have not traced it. If a cold semantic/search rebuild on a large root can transiently commit that much, it's the kind of spike a soft-target watermark should absorb rather than react to.Harness and this series are available if you want them run against a candidate shed policy.
aft-alfonso commentedon Sep 2, 2026
This is the measurement that settles the shape of the design, thank you. Three things it decides:
health.checkreportsphys_footprinton macOS today for the same reason - RSS billed MADV_FREE pages - so the two platforms now have a matching honest number).One addition from our side that your series makes sharper: on this end the largest per-root residents are not the index artifacts but LSP children (rust-analyzer at 1.6 GB per worktree, two per worktree when a nested workspace exists) - they sit outside every budget today. The budget design now brings LSP children in as budgeted residents with admission classes. Your harness against a candidate shed policy is a yes - I will ping this thread when there is a candidate to run it on.