Repository navigation
fleet: cargo target duplication across worktrees — shared compile cache, seeding, and a storage cap #3448
Description
Problem
Every fleet worktree compiles the same dependency graph into its own cargo target dir, and the storage + CPU cost keeps biting:
- 2026-10-26 cleanup: 84 cargo target dirs totalling 1,182.8 GiB deleted from this laptop.
- 2026-11-01 ~01:26 CST: /home hit 100% full (0 bytes free) — the immediate cause of a failed NixOS switch (SQLite fetcher-cache I/O error). Drew ordered a target/cache cleanup; disk recovered to ~55%.
- 2026-11-01 ~03:23 CST, roughly two hours after that cleanup deleted every target dir: 12 fleet worktree
pkgs/id/targetdirs already exist again, 198G total across fleet + primary + worktrees — each one a from-scratch copy of the same ~760-package graph (debug and release profiles multiply it further). - Same night: 12 concurrent lane cargo builds drove load to 118 on 16 cores, and the nix side built the
idpackage twice concurrently from divergent checkout sources (different drv/output hashes for identical intent) — one was a stale Oct-30 toplevel build still holding daemon slots; both had to be found and killed by hand during a production switch. - Nothing is configured today: no sccache (not installed), no
CARGO_TARGET_DIR, no shared cargo config on the laptop.
What to evaluate (measure, then adopt what the data supports)
- sccache (or equivalent content-addressed compile cache) shared across all worktrees — kills the repeated dependency compilation without target-dir locking between concurrent lanes.
- Shared
CARGO_TARGET_DIRper repo across worktrees — proven manually on 2026-10-26 (a worktree pointed at the primary's target reuses the warm cache); the trade-off is cargo's target-dir lock serializing concurrent lane builds. May suit a hybrid: shared for deps-heavy phases, per-lane for final artifacts, if the tooling allows. - Cache seeding for fresh worktrees (reflink/copy-on-write copy of a warm target where the filesystem supports it) so a new lane never starts cold.
- Profile discipline for lanes: build only the profile the task needs (check vs test vs release), instead of several profiles per worktree by default.
- Nix side: a shared binary cache for the fleet's own builds (the
idpackage and the system toplevel) so identical checkouts substitute instead of rebuilding; plus keeping fleet checkouts converged on the same main so drv hashes dedupe in the store. - A storage cap with a janitor: a bounded, automatic pruning rule for target dirs (age/size based, never touching a target with a build in flight) so /home can never silently refill to 100% again — tonight's disk-full broke a production switch.
Acceptance
- Measured before/after on the laptop: total target-dir storage across the fleet after a representative day of lane work, and cold-start build time for a fresh worktree.
- The adopted setup is the default for new fleet worktrees (encoded wherever lanes are provisioned), not a per-lane manual trick.
- No lane workflow regression: concurrent lanes can still build without corrupting each other's artifacts.
Related: #3195 (pkgs/id build-time review — its build-hygiene line names sccache / shared target), and the pkgs/id crate-split issue filed alongside this one (splitting reduces per-build work; this issue removes the duplicated work).
Activity
developing-today commentedon Nov 1, 2026
CODE-1316
dezren39 commentedon Nov 1, 2026
Filed from Drew's question in main chat 2026-11-01 ~03:23 CST ('is there anything in all the targets that we can avoid duplicating'). Triage task 5369 queued at priority 20, DB-verified. Headline numbers are in the issue body: 12 worktree target dirs / 198G regrown within ~2h of a full cleanup; 1,182.8 GiB deleted in the 2026-10-26 precedent.
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done (triage)
Fresh laptop census (2026-11-01 ~09:35Z, this run):
pkgs/id/targetdirs + the primary clone + ad-hoctarget-*dirs — regrown within hours of this morning's cleanup (issue reported 198G at ~03:23 CST).~/git/code/pkgs/id/target67G; worktrees 5197/5140/3688 at 31G/31G/30G. The big worktree targets are debug-profile only (5197: 31G, all underdebug/) — lanes compile debug/test almost exclusively; the dependency graph is what multiplies.CARGO_TARGET_DIRandRUSTC_WRAPPERunset;~/.cargo/config.tomldoes not exist. Nothing is configured, as filed.Evaluation → pinned design:
SCCACHE_DIR,SCCACHE_CACHE_SIZE=50G,~/.cargo/config.tomlrustc-wrapper,CARGO_INCREMENTAL=0(sccache can't cache incremental builds).CARGO_TARGET_DIR— REJECTED as the primary mechanism. One target dir serializes ~12 concurrent lanes on cargo's target lock and thrashes when profiles/flags differ between lanes; the failure mode is lanes blocking each other, which is the resource this fleet is shortest on.target-*), delete those whose worktree's task is terminal / mtime older than 48h, keep the fleet total under a 150 GiB cap (LRU by mtime), never touch the primary clone's target or a live-claimed worktree; dry-run + log on its first runs. This is the safety net that makes another 100%-disk night structurally impossible while sccache reduces regrowth. Packet B is enqueued by Packet A's worker on completion (the queue's fix dedupe key admits one live fix task per issue).Before/after measurements named for the fix packets: Packet A — two fresh-worktree pkgs/id debug builds back to back (wall times +
sccache --show-statshit rate + per-worktree target size) and a re-run of this census total. Packet B — fleet target total after a representative lane day under the cap, and fresh-worktree cold-start time unchanged-or-better.192 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: done
task: 6194 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Triage complete. Verdict posted above; scoped follow-up task 6443 queued (install janitor timer, verify cap, record option 2-5 decisions, close with evidence). No code, branch, worktree, or config changes in this task.
labels: queue/queued, status/needs-triage for live follow-up 6443.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: picked_up
task: 6443 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Claimed atomically; first checkpoint records origin=openchamber-scheduled-lane. Fresh GitHub state: #3448 is OPEN with no hard-stop labels; PR #4175 is MERGED, so there is no active competing PR.
next: Fetch and fast-forward ~/git/code main, inspect the merged janitor install docs, then create the dedicated task worktree and revalidate laptop timer/load state before applying anything.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: progress
task: 6443 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Janitor tests pass (11/11). Installed and enabled the hourly user timer from the updated worktree; systemd unit verification passes (with an unrelated pre-existing invalid proxy-supervisor Wants symlink warning). The first reviewed dry-run measured 372.20 GiB against 150 GiB; the 20-candidate batch projected 266.17 GiB. The bounded apply run completed successfully, reclaimed 106.78 GiB, and reported 276.49 GiB remaining / over_cap=true. More eligible targets remain beyond the 20-target batch; active/open-PR/recent targets remain protected, so the cap is not yet met.
option_decisions:
next: Timer is enabled and active; next scheduled run is 16:03 CST. Re-measure after subsequent bounded hourly passes; keep fleet: cargo target duplication across worktrees — shared compile cache, seeding, and a storage cap #3448 open until under cap or remaining excess is only protected paths. Current issue owner remains dezren39.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: done
task: 6443 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Completed the scheduled-lane close-out. Janitor tests passed (11/11); the hourly user timer is enabled/active and its apply service succeeded. Two bounded applies reclaimed 106.78 GiB and 11.54 GiB. Final dry-run at 2026-11-02T21:54:45Z reports 268.06 GiB / 150 GiB, over_cap=true, with candidates=[]; all remaining paths are protected by the janitor policy (open PR, live task/build, younger than 48h, or unmapped worktree). No protected target was removed.
evidence: PR #4175 merged: #4175 ; dry-run log /home/user/.local/state/cargo-target-janitor/runs/run-20261102T215445Z-1084333.jsonl ; apply logs /home/user/.local/state/cargo-target-janitor/runs/run-20261102T215104Z-1065270.jsonl and /home/user/.local/state/cargo-target-janitor/runs/run-20261102T215426Z-1083022.jsonl. Option 2-5 dispositions are recorded in #3448 (comment).
next: Hourly timer continues re-evaluating protected paths; next scheduled trigger is 16:07 CST. Closing this umbrella under its accepted exception: remaining excess is only on protected paths.