Repository navigation
init: FAILURES=1 forever — sysstat apt-lock collision kept the boot marker unwritten and init re-running every prelude (fixed 2026-11-04) #5700
Description
Summary
Every sandbox boot on 2026-11-04 recorded result=FAILURES=1 from ~/tools/init.sh, and the single failing step was the same in every run of every boot inspected (da56f9ed, ad076b64, d50de26a, 6989413c): the sysstat collector restore, failing instantly with:
E: Could not get lock /var/lib/apt/lists/lock. It is held by process 774 (apt-get)
E: Unable to lock directory /var/lib/apt/lists/
Root cause chain
scripts/sysstat/install-sysstat-units.shrunsapt-get update && apt-get install sysstatwheneversaris missing — which is every boot, because apt packages do not survive sandbox recycles.- The lock holder is the sandbox platform's own post-boot os-intent replay (
apt-get update, pid 774 in boot d50de26a; pid 753 in boot 6989413c, still running 12+ minutes after boot, fetching through the runtime proxy). It holds/var/lib/apt/lists/lockfor many minutes after every recycle. - init counted the collision as a failure →
FAILURES=1→ the boot marker is only written on a clean run, so it was never written. - With no marker,
~/tools/init-ensure.sh(run as the prelude of essentially every agent turn and cron job) took its cold path and re-spawned the full init every time. Observed: init END records every 2–7 minutes all day (boot d50de26a: 20:59, 21:02, 21:05, 21:09, 21:11, 21:14, 21:19, 21:21 UTC). - Each re-run costs ~65 s, dominated by the deferred-telemetry
du -sh /home/hatch/nix-persist(60 s timeout, returnsunknown, every time) — a persistent, pointless background load on a 4-CPU box, plus an apt attempt that could never succeed while the replay held the lock.
All other restore steps (nix, fleet units, charge fetcher, resource sampler, autocommit, backups, airthings) succeed in ~1 s; sysstat was the only failure. The recurring "init FAILURE" records Drew kept seeing were this one step, not a series of different problems.
Fix (landed 2026-11-04, ~/tools/init.sh)
- The sysstat block now captures the installer output. If it failed with
Could not get lock/Unable to lock directory, the run logssysstat restore DEFERRED (apt lock held by another process)and does not count a failure; a detached retry (up to 6 attempts, 5 min apart) converges the collector once the lock frees. - Any other installer failure still counts as a real init failure, so
result=stays truthful. - Verified live in boot 6989413c (2026-11-04 23:11:45 UTC): the first full run under the new code logged
init: sysstat restore DEFERRED (apt lock held by another process — not counted as a failure; detached retry queued)and endedresult=SUCCESS; the boot marker was written for 6989413c andinit-ensure.shnow warm-exits in 0.04 s instead of re-spawning a ~65–97 s init on every prelude. (Two transitional runs that started before the edit landed still showed the old FAILURES=1; the first run started entirely under the new code is the SUCCESS above.)
Notes / follow-ups
- The platform replay holding the apt lock for 12+ minutes is itself worth watching: in boot da56f9ed its fetches were failing (
Connection failed [IP: 198.19.0.1 3128]— the runtime proxy), so on a degraded-proxy boot the lock window can stretch much longer. Nothing fleet-side can shorten it; the defer makes us immune to it. - The 60 s
dutelemetry timeout fires once per init run; with the loop fixed it is once per boot. If it still returnsunknownevery boot, consider dropping the measurement to a cheaper source (df on the mount) — separate small cleanup, not done here. - Related incident record: the 2026-11-04 15:00–17:00 CST responsiveness loss (session proxy CONNECT failures, two failed orchestration passes, a Muse daemon restart at 21:29 UTC, and a sampler-silent window 21:29–22:57 UTC) is filed separately; this init loop was a contributor to baseline load, not the established cause.
Activity
dezren39 commentedon Nov 4, 2026
Fixed and verified 2026-11-04. Fix commit: home repo 563a9fff6 (~/tools/init.sh — sysstat block: apt-lock failures are a logged DEFER with a detached retry, not a counted failure; other installer failures still count).
Verification (boot 6989413c, live): the first init run started entirely under the new code logged
init: sysstat restore DEFERRED (apt lock held by another process — not counted as a failure; detached retry queued)and ended result=SUCCESS (23:11:45 UTC) — the first SUCCESS in any of today's six boots. Boot marker written (6989413c); init-status result=SUCCESS;init-ensure.shnow warm-exits in 0.04s instead of re-spawning a ~65–97s init (with its 60s nix-persist du) on every agent prelude. At verification time the platform's os-intent replay apt-get update (pid 753) had held the lock for 12+ minutes, so the defer path — not a lucky lock window — is what produced the SUCCESS. The detached retry converges sysstat once the lock frees.Left as notes on the issue (not blockers): the once-per-boot 60s du telemetry timeout, and watching the platform replay's lock duration on degraded-proxy boots.
dezren39 commentedon Nov 5, 2026
WS-C: init itemization + cuts — the ~65–75s was ~70s of telemetry, ~3s of actual restore work
Reconstructed from
~/logs/init.logfor boot6989413c(7 full init runs on 2026-11-04, init_secs 68–97, last run 75s SUCCESS) plus standalone re-measurement of every component on 2026-11-05.Timing table (per full init run)
nix evalnix-ensurenix-wrapper,nix-profile-sync,bash-wrappergo-check,rust-check,services-verifytimeout 60 du -sh ~/nix-persistunknownresults in log history, 7/7 todaynix --version,ls /nix/store, profile count, quota querysystemctl is-active, so ~0 on re-runsNote:
init_secondsis computed before the unit-restore blocks run, so the cold-run ~10s of restores isn't even in the headline number. Also: quota-recorder units are not restored by init.sh at all (no reference anywhere in the init path) — they currently survive via another mechanism; flagged, not changed.Cuts implemented — dezren39/hatch
170dde9e5(tools/init.sh, pushed to main)duoff the init path: measured detached into~/logs/.nix-persist-size; init reports the last completed measurement. The value is telemetry-only, nothing gates on it; same defer pattern as the sysstat apt-lock fix. Saves the full 60s.Before / after (measured)
init.sh(boot marker set aside; init is idempotent by design): init_seconds=9, result=SUCCESS, marker rewritten,net_lat_s=2.445recorded correctly,init-ensurewarm exit 0.21s. Log-verified in init.log (steps 3s + telemetry 6s).Not implemented (recommendations)
services-verifyis a paid-for no-op (~0s) — leave it; it's the intended re-enable point for the disabled services.dustill burns background I/O for a few minutes post-init; if even that matters, drop the measurement entirely — the field has beenunknownfor the entire recent history with no observed consumer harm.Unverified until the next natural boot
Cold-path behavior under real recycle load (os-intent apt replay contention, cold caches): the after-number above is a warm manual run on the current boot. First cold run will also report
nix_persist_size=unknownonce (cache file won't exist yet on a fresh home state if logs are wiped), then real values thereafter — the detacheddufrom today's manual run was still running at filing time, which itself confirms why it never belonged on the init path.