Repository navigation
pkgs/id: split the monolith crate into sub-crates along module seams (prioritized by Drew 2026-11-01) #3447
Description
Order
Drew, 2026-11-01 (~03:23 CST, main chat): "prioritize the crate splitting work."
This elevates the crate-shape work out of #3195's sequencing. #3195 scoped splitting as "only if the above is insufficient" (after profile separation, release audit, build hygiene). Drew's order makes the split a first-class work item now, in parallel — the profile work in #3195 continues on its own track and is not a precondition.
Why
pkgs/id is one giant compilation unit and the cost is now gating production work:
- pkgs/id: build-time review — runtime performance first, then build time (cold check build ~16 min, tests run 12 s) #3195's table: 150 src .rs files, 125,722 lines (2026-10-31), up from 71,723 lines on 2026-10-11 — compile time tracks the crate's own size, superlinearly under LTO with
codegen-units = 1. - pkgs/id: build-time review — runtime performance first, then build time (cold check build ~16 min, tests run 12 s) #3195 measured a cold nix build of the package at ~16 min under shared load (2026-10-31).
- 2026-11-01: the NixOS switch that deploys the fleet-feed collector (pm-stack NixOS module: fleet-feed collector units are hand-written — fold service + timer into nixos/services/pm-stack #3363) sat behind this compile for over an hour of wall time — the
idrelease build (cargo, ~200-crate dependency graph, LTO) ran 53+ minutes under fleet load, and two stale/duplicate builds of the same package were found competing with it and had to be killed by hand. - Every check/test build recompiles the one crate end-to-end for any change anywhere in it. Splitting along the existing module seams makes most changes recompile only the touched sub-crate and lets the graph build in parallel.
Scope
- Identify the split seams from the crate's existing module structure and dependency directions (the program's areas — artifact, world, sandbox, web, dht/ssh, core types — are the starting hypothesis; the triage verifies against real import graphs).
- Sequence the extraction into phased, independently mergeable packets (one sub-crate per packet), each keeping the full test suite green.
- Keep the public surface of the resulting top crate/binary unchanged for consumers (the NixOS module, mcpx, and the flake checks build the same binary).
Acceptance (inherited from #3195, non-negotiable)
- Shipped-binary runtime performance unchanged-or-better, with evidence: the deterministic fuel baselines in
src/sandbox.rsidentical after each phase; runtime spot benchmark before/after. - Cold check-build time materially reduced, before/after timings recorded on this issue from the laptop.
- No test coverage removed and no check weakened to buy speed.
Program context: #3195 (build-time review), Drew's 2026-10-28 id program directive. Sibling storage/duplication problem: the fleet target-dir duplication issue filed alongside this one.
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Large: multi-day or open-ended; expect staged check-insImportant - do this very soon, before anything not expeditedImportant and time-sensitiveFiled by the new-issue-creator jobTriaged and routed (see agent note)Queue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Nov 1, 2026
CODE-1315
dezren39 commentedon Nov 1, 2026
Prioritized by Drew in main chat 2026-11-01 ~03:23 CST. Triage task 5366 queued at priority 10 (top band) on repo-health-code-pilot, DB-verified. The triage maps the seams and routes phased extraction packets; #3195's acceptance (fuel baselines identical, runtime perf neutral-or-better, no coverage loss) binds every phase.
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done (triage)
Module map at current main
70c206250— pkgs/id/src: 150 .rs files, 126,981 lines, single crate (no workspace yet). Cross-reference counts are files in area A referencingcrate::<module>of area B (laptop count, this run):Pinned sequence (each phase one independently mergeable packet, full suite green, binary surface unchanged; each phase's worker enqueues the next phase on completion, reusing the standard fix dedupe key once the prior task is terminal):
crate::paths. → task 5371 (enqueued).Timing method (from #3195, required in every phase packet): cold
nix buildof pkgs/id (flake check phase) wall time on the laptop, before/after, plus coldcargo checkwith an emptyCARGO_TARGET_DIR; record both on this issue. #3195 baseline: ~16 min cold nix build under shared load, 1,963 lib tests executing in 12.4 s — the cost is this one unit's compilation.Explicitly NOT in scope for any phase: behavior/API/CLI/wire changes, test weakening, profile work (#3195's track).
155 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · done (triage)
Findings — the split is underway, not unstarted:
pkgs/id/Cargo.tomlon main is now a workspace root (members = ["crates/id-core"], vendored p2panda crates excluded) and its header states: "Phase 1 of the pkgs/id crate split (pkgs/id: split the monolith crate into sub-crates along module seams (prioritized by Drew 2026-11-01) #3447): the leaf modules live in the id-core sub-crate under crates/".pkgs/id/crates/id-core/src): access, connectivity, discovery, membership/, petname, qr, sync_method, time, tuple, unicode, waiting, world_caps, world_limits, world_presentation.pkgs/id/src, still the bulk of the ~309 .rs files under pkgs/id): artifact, sandbox, store, event_store/event_sync, groups/, dht, frames, hosted_binding, server_config, repl/, and ~50 more top-level modules — owned by steps 2–7 above in that dependency order.Routing: no new issues or tasks needed — the step issues are the fix path and are already queued. This issue stays OPEN as the umbrella until step 7 (#4411) lands. Per-phase acceptance evidence (sandbox fuel baselines identical; before/after cold check-build timings recorded here) remains owed by each step's PR, per the acceptance block on this issue.