Repository navigation
pkgs/id: build-time review — runtime performance first, then build time (cold check build ~16 min, tests run 12 s) #3195
Description
Priority order (Drew, 2026-10-31)
- Optimal runtime performance of the shipped binary — first, non-negotiable. The review should also verify the current release profile is actually optimal by current Rust best practice; if a change makes the shipped binary faster, that is in scope and welcome.
- Build time — second, and wanted too. Reduce check/test/CI build times as far as possible without compromising (1). Both outcomes are the goal, in that order.
Measurements (2026-10-31, laptop)
- Cold nix build of pkgs/id (the flake check phase, compile + test): ~16 min wall (10:25→10:41 CDT), under shared load. The 1,963 lib tests themselves executed in 12.4 s — the time is essentially all compilation.
- Verifying the pkgs/id: 5 lib tests failing on main block the NixOS system build (rusqlite guard, fuel budgets, world_ws Screen drift) #3160 test fixes the same day: a single
rustcinvocation for theidcrate ran 26+ min under contention (CPU counters sampled live: a full core consumed continuously — it is codegen, not a stall). - The cost gated a production deploy: the NixOS system build failed on pkgs/id's check phase and the fix-verify loop is dominated by this compile.
Why it keeps getting worse
The crate is one giant compilation unit and it is growing much faster than its dependency tree (git, pkgs/id):
| date | src .rs files | src lines | Cargo.lock packages |
|---|---|---|---|
| 2026-10-11 | 76 | 71,723 | 705 |
| 2026-10-20 | 132 | 104,507 | 752 |
| 2026-10-31 | 150 | 125,722 | 762 |
Everything, including test/check builds, inherits the release codegen configuration (thin LTO, codegen-units = 1) — the slowest settings — and every test harness links wasmtime + iroh. Compile time tracks the crate's own size, superlinearly under LTO.
History
#869 (closed) was this problem at the CI level — "id flake check takes ~51 min and dies at the job timeout" — resolved by raising the timeout 60→90 min (#870), i.e. by accommodating the cost, not reducing it. This issue is the follow-up that reduces the work.
Scope — evaluate against best practice, measure, adopt what the data supports
- Profile separation:
[profile.test]/ check builds should not inherit LTO +codegen-units = 1from the release profile. This is the standard Rust practice and is expected to be the largest single win for the check path; it changes nothing about the shipped artifact. - Release profile audit (for priority 1): thin vs fat LTO, codegen units, panic strategy, linker choice,
target-cpuportability trade-offs, PGO feasibility — adopt only changes with benchmark evidence. - Build hygiene: linker (mold/lld), caching (sccache / shared target), incremental settings for local dev builds, workspace profile placement.
- Crate shape (only if the above is insufficient): splitting the crate or moving example/test code out of the lib harness.
Acceptance
- Release binary optimization is unchanged-or-better, with evidence: fuel baselines in
src/sandbox.rsidentical after the change (they are deterministic — any drift means behavior changed), plus a runtime spot benchmark of the shipped binary before/after. - Cold check-build time (nix build of the id package, and
cargo test --no-runin a clean target dir) materially reduced, with before/after timings recorded on this issue from the laptop. - No test coverage removed and no check weakened to buy speed.
Program: Drew's 2026-10-28 directive (full text: hatch-personal memory/topics/id-program-2026-10-28.md).
Working pattern: parallel investigations, then the named synthesis issue merges them into a full design. Route guidance: Workhorse by default; Specialist/Summit for synthesis and ramification-heavy work. A result that says "done" is not the endpoint — the orchestrator reviews it and may commission another round. Deliverables: a markdown design/investigation doc (mermaid/SVG diagrams welcome) linked from this issue, plus follow-up issues for anything buildable. Build a proof of concept where the design calls for one, but do NOT build the permanent feature on top of an unreviewed PoC.
Acceptance criteria
- A review is recorded on the issue covering, in the issue's priority order: (1) runtime performance of the shipped binary — including verification that the release profile matches current Rust best practice — and (2) build/check/test time, with before/after measurements for any change proposed.
- Any profile or build change adopted keeps shipped-binary runtime performance neutral-or-better (measured), per the issue's non-negotiable ordering.
- Follow-up implementation work the review identifies is filed as scoped issues rather than left in this review's comments.
Activity
dezren39 commentedon Oct 31, 2026
Direction update (Drew, 2026-10-31) — the crate split is now the priority
Drew's ruling, superseding the scope ordering above: the crate split is the priority approach, not a fallback. He is open to completely refactoring the entire structure of pkgs/id.
What this means for the review
First deliverable: a design (per the program's working pattern)
A design doc proposing:
Sequencing note
Design against the tree after #3160 merges — the five-test fix is editing
sandbox.rs,commands/world.rs,web/world_ws.rs, andstore_traits.rsright now, and it remains the critical path (it gates the NixOS deploy).dezren39 commentedon Oct 31, 2026
Intake review
Fleet monitor intake pass (protocol §5), 2026-10-31.
Related: Build/test-health set for pkgs/id: code#3160 (5 failing lib tests), code#3199 (load-racy log_sync tests). History: #869 (closed) was the CI-level version of this problem.
Notes / risks / suggested approach: Still live: measurements in the issue are from 2026-10-31 and no merged PR resolves this review as of this run. Notes: the crate's growth table (76 → 150 src files in 20 days) means build time worsens on its own; thin LTO + codegen-units=1 inherited by test builds is the named suspect. Risks: build-time fixes that slow the shipped binary violate the issue's ordering — measure both. Suggested approach: start with profile split (test/check profile without release LTO) and verify runtime benchmarks before touching the release profile.
dezren39 commentedon Nov 1, 2026
Two updates from main chat 2026-11-01: (1) Drew ordered the crate-splitting work prioritized — it is now first-class as #3447 (triage task 5366, priority 10), superseding this issue's 'only if the above is insufficient' sequencing for the split; the profile/audit/hygiene tracks here are unaffected. (2) This review finally has a queue task: triage 5367 at priority 10, DB-verified. Also: your recorded cold nix build of the id package (~16 min, 2026-10-31) is now the fleet's baseline figure — tonight's switch-gating build ran 53+ min under load ~118.
452 remaining items
dezren39 commentedon Nov 2, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
verifiedcheckpointdezren39 commentedon Nov 2, 2026
🤖 Agent note · parked — laptop load gate unmet
world_benchmedians at 15-minute load < 20; the laptop never met the gate this shift (best 20.71/20.82 at 18:06Z; 54.07/47.41/36.22 at 18:23Z with ~49 runnable threads from other fleet cargo jobs). Numbers taken at load ~50 would not be comparable tobenches/BASELINES.md.~/git/fleet/5378-3195-release-audit(branchqueue/5378-3195-release-audit, at c91c679) withrun-audit.shrunning detached (setsid): variants (a) baseline thin-LTO/cg1, (b) fat LTO, (c) cg16, (d) lld — per-variant target dirs, build seconds + bench lines logged toaudit-5378.login the worktree. Baseline build started 18:14:01Z and was still compiling at park time.audit-5378.log; the built bench executables persist undertarget-<variant>/release/deps/world_bench-*— re-run the executables themselves under the load gate before using any figure measured at load >20, then post the measured table + recommendation on issue pkgs/id: build-time review — runtime performance first, then build time (cold check build ~16 min, tests run 12 s) #3195. No [profile.release] change in this task.dezren39 commentedon Nov 2, 2026
agent-note:v1
event: picked_up
task: 5378 (repo-health-code-pilot; OpenChamber scheduled lane; lane 9aca43b4-5439-4939-9f71-323f5d32521b)
state: claimed; resuming audit-runner checkpoint.
verified: Fresh checks at 2026-11-02T19:24Z: issue #3195 OPEN, no hard-stop labels; phase-1 PR #3674 is now MERGED (was OPEN); no open phase-2 PR. Laptop worktree ~/git/fleet/5378-3195-release-audit exists; audit-5378.log shows variant a-baseline complete (build 2072s, bench 207s) and variant b-fat build complete (1807s) with bench exe now running. Load 6.16/15.87/22.86. Runner process live. No profile change made.
next: Checkpoint progress and park; runner continues detached. Resume collects remaining variants and posts the table.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: parked
task: 5378 (repo-health-code-pilot; OpenChamber scheduled lane; lane 9aca43b4-5439-4939-9f71-323f5d32521b)
state: parked until 2026-11-02T19:46:40Z (20 minutes).
progress: baseline completed: build 2072s and bench 207s; medians journal_append_fsync 10789.033us, checkpoint_compaction 18473.069us, guest_replay 0.291us, session_fanout 0.145us. b-fat build complete (1807s); benchmark child PID 4154379 still running. Existing parent PID 3580824 is SIGSTOPed so it cannot launch remaining unguarded builds; resume must collect/re-run b-fat as appropriate, prevent old parent from starting c/d, and run remaining variants under ~/tools/fleet-build-lock. Issue #3195 OPEN; phase-1 PR #3674 MERGED; no phase-2 PR. No profile change.
resume: Continue the partial audit and post the measured table/recommendation on #3195; record load for each run.
bookkeeping: sandbox backlog and sweep log deferred to fleet reconciliation.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: progress
task: 5378 (repo-health-code-pilot; OpenChamber scheduled lane; lane 9aca43b4-5439-4939-9f71-323f5d32521b)
verified: Follow-up at 2026-11-02T19:33Z: b-fat benchmark child has exited; audit-5378.log now contains medians journal_append_fsync 17352.813us, checkpoint_compaction 18350.810us, guest_replay 0.206us, session_fanout 0.107us. Exact load during that benchmark was not logged, so treat these as provisional and not controlled comparison data.
resume detail: parent PID 3580824 remains SIGSTOPed; do not CONT it directly because it would start c-g16/d-lld without fleet-build-lock. Resume by safely retiring the old parent and running remaining variants under ~/tools/fleet-build-lock; verify/re-run b-fat if needed. No profile change.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: picked_up
task: 5378 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151)
at: 2026-11-02T19:47Z
state: claimed; fresh verification: #3195 OPEN, phase-1 PR #3674 MERGED, no open phase-2 PR, no hard-stop labels. Existing checkpoint has baseline metrics, b-fat build time, stopped runner PID 3580824, and b-fat benchmark PID 4154379; current process scan shows the benchmark child defunct.
next: inspect the existing worktree/log and lock script; resume only recorded remaining work under fleet-build-lock. No profile change made.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: parked
task: 5378 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151)
at: 2026-11-02T20:00Z
state: parked for 30 minutes after fresh verification: #3195 OPEN, phase-1 PR #3674 MERGED, no open phase-2 PR, no hard-stop labels.
partial evidence: existing baseline (build 2072s, bench 207s) and fat-LTO (build 1807s) medians are preserved in the task worktree but were gathered under heavy load, so they are provisional. Re-runs did not obtain a build-lock slot before this claim window ended.
resume: use /home/user/git/fleet/5378-3195-release-audit (base c91c679); do not resume old run-audit.sh (terminated). c-g16 PID 80133 and baseline FUEL_BUDGETS test PID 95809 are running under ~/tools/fleet-build-lock; baseline/fat repeat PID 89706 and d-lld PID 119104 are waiting for the lock. Check the recorded output paths and audit-repeat-20261102b.log; finish the full table and recommendation before deciding on any packet. No source/profile changes.
PGO feasibility: llvm-profdata and lld are installed; expect at least two ~34.5-minute release builds plus ~207s training and merge, and current guest_replay measures host-side replay rather than actual WASM sandbox behavior.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: progress
task: 5378 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151)
at: 2026-11-02T20:11Z
state: parked
update: after the queue claim was parked, the fleet-build-lock c-g16 build completed successfully in 11m15s (slot 2/2, CARGO_BUILD_JOBS=8). Executable: /home/user/git/fleet/5378-3195-release-audit/target-c-cg16/release/deps/world_bench-04adad38f1012e24. No c-g16 benchmark has run yet, so there are no medians.
resume: run this executable under fleet-build-lock and continue collecting the remaining audit outputs. Queue checkpoint could not be changed because task 5378 is parked and no longer claimed by this worker; queue state remains parked as recorded.
dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 2, 2026
🤖 Agent note · done
Phase 2 release-profile audit — results (task 5378)
Machine: laptop (amd-6), same machine/session for all clean runs, base c91c679, worktree
/home/user/git/fleet/5378-3195-release-audit, logsaudit-5378.log(original heavy-load run),audit-clean-20261102.log,fuel-repeat-20261102.log. Bench =benches/world_bench.rs, medians of 5 reps, µs. Load gate (15-min avg < 20) was met for the first time this session at 20:37Z; the original 18:14Z runs (load 39/36/28) are superseded and not used below except for cold build times.Margin threshold (stated up front): a variant earns an implementation packet only if it beats baseline by ≥10% on the CPU-bound benches (guest_replay, session_fanout) without regressing checkpoint_compaction. None did.
world_bench medians (µs, lower is better)
Reading: journal_append_fsync is disk/load-dominated (baseline-level values appear for fat LTO in the repeat once load settled) — it does not separate the variants. The CPU-bound benches do: fat LTO is consistently worse (guest_replay +198%/+295% vs baseline, session_fanout +68%/+101%, compaction +69%/+227%). codegen-units=16 is mixed-to-worse (fanout −6%/−14%, but guest_replay +81% in both runs and compaction +75%/+77%). lld is within ±2% of baseline on all four benches — runtime-neutral, as expected (identical codegen, different linker).
Release build times
Oracle
cargo test --release --lib checked_in_examples_stay_within_fuel_budgetson the baseline build: ok, 1 passed (1.33s test time; FUEL_BUDGETS in src/sandbox.rs unchanged and green).PGO feasibility
Feasible tooling-wise: llvm-profdata and lld are installed. Cost: at least two extra release builds (~35 min each at the baseline build time) plus a training run (~207s at bench scale) and profile merge. Representativeness problem: world_bench's guest_replay measures host-side replay, not actual WASM sandbox guest execution, so a profile trained on these workloads may not transfer to the shipped workload. Not recommended without a representative training workload.
target-cpu=native
Not measured this run (optional data point only). It is not shippable via nix and is labelled as such.
Recommendation
No variant earns an implementation packet. Keep the current release profile (thin LTO, codegen-units=1, default linker). Fat LTO regresses the CPU-bound benches; codegen-units=16 trades a small fanout gain for large guest_replay/compaction regressions; lld is runtime-neutral (revisit only as a build-time change with a clean cold-build comparison, which this audit does not establish). No [profile.release] change was made by this task.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · re-triage verdict — phases 1–2 landed/done; phase 3 re-enqueued with a corrected packet
Fresh state (2026-11-03,
gh+ queue DB): issue #3195 is OPEN, priority/high, no hard-stop labels, no open PR. Per phase:Disposition: this issue stays OPEN pending task 5375 (phase 3 measurement) and the #3447 step timings. Its
queue/claimed,status/in-progress, andwork/assignedlabels remain — they now describe task 5375's live claim of the follow-up packet.Checks a follow-up must run: cold timing runs under fleet-build-lock as named in the task 5375 packet; id test suites via CI on any adoption PR, with FUEL_BUDGETS byte-identical.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · blocked (pre-existing failure on main)
pkgs/idwith--all-targets. No config change made; [profile.release] andFUEL_BUDGETSuntouched.What was run (laptop amd-6, worktree
~/git/fleet/5375, branchqueue/5375-3195-mold-measure, basec0e9d059a)Under
fleet-build-lock(slot 1/2,CARGO_BUILD_JOBS=8), fresh target dir per run,cargo check --profile ci-check --all-targets:-C link-arg=-fuse-ld=<nix mold>)The elapsed times are times-to-failure on a non-compiling tree, not cold check-build timings, and no mold conclusion can be drawn from them.
The blocker (pre-existing, linker-independent)
Both runs fail identically:
ChainExt(pkgs/id/src/chain.rs) gainedpub checkpoint: Option<SessionCheckpoint>for #3794 (PR #4627, merged today), but integration-test literals were not updated. Filed as #4850 with the full evidence and done-when.Resume condition
After #4850 lands on main: re-run this same protocol (cold, fresh target dirs, default vs mold 2.42.1, load recorded per run, fleet-build-lock) and record the real timings here; adopt mold for check/dev profiles only on a material cold win. The worktree and branch above are clean (only the measurement script untracked) and can be reused or recreated.
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 5375 (small_fix, blocked, priority 40) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03