Repository navigation
Init job duration: ~60s of every ~75s run is a telemetry du of nix-persist that times out and returns 'unknown' #5717
Description
Init job duration breakdown — why every init run costs ~65–97s (measured 2026-11-04)
Follow-on from #5700 (the re-run loop, fixed). The loop is fixed, but each individual run of ~/tools/init.sh still costs 68–97s (boot 6989413c runs: 71/68/73/97/79/75s). Drew asked where the time goes. Per-step timings from ~/logs/init.log + ~/logs/init.jsonl, cross-checked against ~/tools/init.sh:
Ranked breakdown (final SUCCESS run, 23:10:30→23:11:45 UTC, init_secs=75)
| # | Step | Seconds | Why it's slow | What would cut it |
|---|---|---|---|---|
| 1 | du -sh /home/hatch/nix-persist (deferred-telemetry block, init.sh:207) |
~60 (the full timeout 60) |
Recursive du over the whole persisted nix store tree (15,247 store paths) on a loaded 4-CPU box. It times out and returns unknown in every recent run — the value is never even obtained. ~80% of the entire init cost buys a telemetry field nobody reads. |
Delete it, or cache it (compute at most once per boot into a state file), or move it to the detached-retry pattern the sysstat restore already uses. Any of these cuts init to ~15s. |
| 2 | Quota snapshot (/opt/hatch/bin/subscription-status status, init.sh:220, timeout 30) |
~2–10 (varies; budget 30) | Network/API call on every init run. | Cache per boot; it is telemetry, not a restore step. |
| 3 | Net check (init.sh:210–216) | ~3–4 | 1 HEAD + 3 full GET samples to cache.nixos.org through the CONNECT proxy (net_lat alone averaged 2.2s). | One sample suffices for a liveness check. |
| 4 | Snapshot counts (ls /nix/store | wc -l, profile bins, init.sh:218–219) |
~1–3 | Directory listing over 15k entries. | Negligible alone; cache with #1 if kept. |
| 5 | All named restore steps combined (nix-ensure 1–2s, nix-profile-sync 1s, nix-wrapper/bash-wrapper/go-check/rust-check/services-verify 0s each) | ~4 total | — | The actual restore work is already fast. Nothing to fix. |
| 6 | sysstat restore (init.sh:280–306) | 0s when apt lock held (deferred instantly, post-#5700 fix); minutes when the lock is free | Its apt-get runs inline under a 600s budget; the platform os-intent replay's own apt-get update fetched 41.7 MB in 10m33s at ~66 kB/s through the proxy on this boot. On a boot where init wins the lock race, init eats that cost inline. |
Always detach (never run apt inline in init); the deferred-retry path already converges it. |
The amplifier (context, fixed by #5700 but worth restating)
While FAILURES≥1 kept the boot marker unwritten, init-ensure re-spawned this full ~65–75s init on every agent prelude, all day, every boot — the ~65s cost was paid hundreds of times per day, and the log shows overlapping invocations serializing (another init already running, exiting). Any future FAILURES≥1 cause re-arms that herd at ~75s per prelude. Shrinking the run (items 1–3) shrinks the blast radius of any recurrence; a rate-limit on init-ensure respawns would cap it independently.
Bottom line
Init's real work is ~4s. ~60s is a single telemetry du that times out and returns unknown. Removing/caching it (plus detaching the sysstat apt path unconditionally) takes a run from ~75s to ~10–15s.
Evidence: ~/logs/init.log lines 15997–16016 (two full runs, step-stamped), ~/logs/init.jsonl (424 runs), ~/tools/init.sh lines 199–306. Filed from the 2026-11-04 laptop-session coordinator run (Phase 1 addition at Drew's request).
Activity
developing-today commentedon Nov 5, 2026
CODE-1417
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
12 remaining items
dezren39 commentedon Nov 5, 2026
🤖 Agent note · picked up
~/tools/init.sh, not a tracked file in this repo; prior lane could not comment due GitHub 403 secondary rate limit. I will finish the triage write and route the access boundary.dezren39 commentedon Nov 5, 2026
🤖 Agent note · triaged
cc45860abb4427fa873f26b81583682af2a7d548, the repo has no tracked~/tools/init.shornix-persistreferences; this laptop’s/home/user/tools/init.shis absent. Fresh GitHub state: Init job duration: ~60s of every ~75s run is a telemetry du of nix-persist that times out and returns 'unknown' #5717 OPEN, no hard-stop labels, no matching open PR. The measurements in the issue body report thatdu -sh /home/hatch/nix-persistconsumes the full 60s timeout and always yieldsunknown.dutelemetry (it is not providing a value); if retained, compute at most once per boot or defer it. Keep the apt/sysstat work detached from init as the issue body describes. Implementing/testing requires access to the host-owned~/tools/init.shor a host-level queue/worktree; neither is available in this code-repo lane. No repo PR or code changes.dezren39 commentedon Nov 8, 2026
agent-note:v1
state: done
task: 12266
worker: repo-worker-code
lane: code-h
issue: #5717
summary: Re-triage (task 12266, direct lane) under the triage doctrine. Verdict: NO LONGER REPRODUCIBLE — the headline cost was removed from the live init script, verified against fresh runtime logs. Closing as an explicit verified closure with evidence.
Evidence (fresh, 2026-11-08, sandbox host where ~/tools/init.sh actually lives — this was never a tracked file in developing-today/code, as the task 10145 triage already established):
3a02ea3to6b71c24#3) was also improved in the same script: the 3 latency samples now run in PARALLEL (comment at the snapshot block: sequential samples summed to ~22s of the 97s run on 2026-11-04).Disposition: issue #5717 closes with this verified evidence; task 12266 completes as no_longer_reproducible. No fix packet is created — there is no remaining real work scoped to this issue.