fleet step 2: queue domain pilot — brokers into pkgs/fleet, one-flip-at-a-time migration #5430

Activity

1. dezren39

   added

   kind/enhancement

   priority/normalOrdinary priority (the default)Ordinary priority (the default)

   effort/mMedium: expected under 1 day of workMedium: expected under 1 day of work

   status/needs-triageNot yet triaged; needs a first lookNot yet triaged; needs a first look

   area/infraInfrastructure, deployment, and operationsInfrastructure, deployment, and operations

   on Nov 4, 2026
2. dezren39 commented on Nov 4, 2026

   

   dezren39

   on Nov 4, 2026

   Member

   More actions

   Intake review

   🤖 Agent note · state: filed

   • Agent: fleet monitor
   • Session / ID: repo-fleet-monitor run 2026-11-04 06:12 CST
   • Found via: monitor reconciliation
   • Task: Intake enrichment (protocol §5): verified still live in a bounded check this run (no merged PR or closed duplicate naming this fix), appended acceptance criteria, normalized labels.
   • Expectation: as labelled (effort class on the issue); done = acceptance criteria met with evidence linked here.

   Related: epic #5412 and steps #5429-#5433 form one set; #5413 is a sibling absorption task. Native sub-issue attachment per protocol §11 is recommended for the epic if not already done.
   Risks: Sequencing: the fleet steps are gated in order (notably #5433 on the #5350 step 6 cutover); starting out of order risks rework.
   Suggested approach: Triage through the queue in dependency order, verifying each step's acceptance evidence before closing.
3. dezren39

   added

   meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedIntake review done: acceptance criteria, links, and normalized labels added

   on Nov 4, 2026
4. dezren39

   added a commit that references this issue on Nov 4, 2026

   Merge pull request #5478 from developing-today/fleet-step2-queue

   ...

   cefda11
5. dezren39 commented on Nov 4, 2026

   

   dezren39

   on Nov 4, 2026

   Member

   More actions

   Step 2 complete — queue domain landed and live

   PR #5478, merged as cefda118e0d4143867b73c80ef72eec09a78e370 (merge commit on main; the merged tree is byte-identical to the build verified below — git diff across every step-2 file: 0 lines). Because main had moved (fleet-secrets hardening #5472 among others), the branch was rebased onto current main and everything was rebuilt and re-verified on the rebased tree before merge: nix builds green, smoke 42/42 again, and a fresh scratch-stack round-trip (task 9555). After the merge, the live units and the sandbox binary were converged to the merged build's artifacts (same procedure, health + round-trip re-verified: task 9559). Final live artifacts: units ExecStart /nix/store/mhjkhyw8d8pmvd39awm6z05h81r00g29-fleet-0.1.0/bin/fleet serve <role>; sandbox binary from /nix/store/qby1k3d00w1dkz8f40k9a9bxq46prrj5-fleet-static-static-x86_64-unknown-linux-musl-0.1.0, sha256 278f9f760140cf5b1c12caead6a7692b3244128bcd15a9fce033018c6b4e095b. The flip-by-flip record below cites the artifacts in force at each flip. The broker family now lives at pkgs/fleet/crates/fleet-queue/; the laptop's queue stack runs on fleet serve <role>; the sandbox repo-queue resolves to the fleet artifact. All three flips were performed one at a time on 2026-11-04, each gated on health plus a real enqueue → claim → complete round-trip through the flipped component before the next began. Flips waited out the fleet-switch window (attempt 4's activation had finished — SWITCH_EXIT=4, a libvirtd failure unrelated to the queue — before flip 1).

   The port

   • Client (1,158 LOC), queue-broker, secrets-broker, and validate.rs (verbatim) moved from the sandbox workspace into crates/fleet-queue, refactored onto the platform crates: queue-broker credentials resolve through a fleet_secrets::Resolver whose source speaks the secrets-broker HTTP API; the secrets-broker refresh runs bws through the CommandRunner seam; one logging facility (src/log.rs, {domain}: {msg} — byte-identical to the ad-hoc prefixes). Wire behavior preserved (envelopes, framing, exit codes; serde_json/preserve_order where consumers read key order).
   • repo-queue gains --help (rendered from the registry) and --version (argv0 route) — the census gap. fleet queue … is identical.
   • One platform addition the port required: SecretValue::from_resolved — step 1 shipped the newtype with no constructor usable by sources outside the crate.
   • One deliberate behavior change: secrets-broker refresh errors no longer embed a trimmed bws stderr excerpt (the CommandRunner contract carries exit status only). Errors carry names/status, never values.

   Flip evidence

   Scratch queue fleet-step2-pilot throughout; synthetic packets (issue numbers 999999001–999999008) so no production queue was touched. Baseline on the untouched stack first: task 9537 round-tripped. Pre-flip proofs: fleet brokers on scratch ports against live services (114 secrets refreshed; full round-trip, task 9544).

   1. secrets-broker — unit ~/.config/systemd/user/secrets-broker.service: ExecStart %h/fleet-brokers/bin/secrets-broker → /nix/store/y2g8ddkwsqx3xi8f0c64aisf2c4jvs6s-fleet-0.1.0/bin/fleet serve secrets-broker (pre-rebase build; converged to the merged build afterwards — see the header). Process exe read back from /proc; journal: listening on 100.99.113.103:4800, refreshed 114 secrets. Verified: /health (114 secrets), /secret/supabase.muse-dev2.project-url shape (200, key match, non-empty value — never printed), and a round-trip whose queue-broker credentials were served by the flipped broker (task 9546 → done).
   2. queue-broker — unit ExecStart → …/bin/fleet serve queue-broker. Journal: listening on 100.99.113.103:4801; /healthz ok. Verified: full round-trip from the sandbox consumer client via the session proxy (task 9547 → done); the laptop claimant's read path (rich list on repo-health-code-pilot, queued + claimed) served by the flipped broker; queue counters kept churning post-flip (claimed 220→212, parked 50→57 in the first minutes).
   3. client symlink — sandbox ~/tools/bin/repo-queue: /home/hatch/tools/repo-queue → /home/hatch/tools/fleet, a copy of the fleet-static artifact (full profile, musl static — a glibc-dynamic laptop build names store paths the sandbox lacks; linkage per context is refinement B), sha256 verified both sides on transfer (pre-rebase build 488c20db…; converged to the merged build 278f9f76… afterwards). Verified: repo-queue --version → repo-queue 0.1.0, --help renders, health answers, round-trip task 9548 → done.

   Rollback — exercised, not assumed

   • The in-repo standalone stack (fleet-brokers package: repo-queue 510,344 B / queue-broker 1,980,960 B / secrets-broker 616,944 B) ran a complete scratch stack and passed the same round-trip (task 9549 → done).
   • Unit rollback artifacts kept in place: ~/.config/systemd/user/{secrets,queue}-broker.service.pre-fleet-step2 (pre-flip files), original binaries untouched in ~/fleet-brokers/bin/, sandbox original at ~/tools/repo-queue (flip = one ln -sfn back). The original client still reads the flipped broker fine.
   • Sandbox source tree archived in place with a SUPERSEDED.md; laptop ~/fleet-brokers/SUPERSEDED.md marks the binaries as rollback inventory.

   Home module

   home/common/default.nix now defines both units as fleet serve <role> from this flake's pkgs/fleet (PR #1459's pattern, per D1) plus the fleetPkgs binding (atlas-search toolchain pattern). The live hand-written files were re-pointed in place; home-manager converges them on the next activation.

   What the census/design got wrong about the live queue

   1. Claim requires a registered worker. tasks.worker_id has an FK to a workers table: claiming with an unregistered worker id fails with PostgREST 409 (23503). The pilot round-trips claim as the registered default worker repo-worker-code.
   2. Packet validation is much richer than the note implies — the manual schema requires problem/priority_reason, non-empty evidence/acceptance/tests, scope include/exclude, risk, timebox, max_attempts, a dedupe block, and an identity (issue/prs/set/source), plus dedupe-key family↔kind↔number consistency. All preserved verbatim in the port.
   3. /health is intermittently unavailable platform-wide: the repo_queue_health view query hits PostgREST statement timeouts (57014) under load — reproduced identically on the pre-fleet broker minutes before flip 2, so it is a Supabase-side condition, not a port defect. The broker envelope behavior matches byte-for-byte; the load-bearing paths (enqueue/claim/complete/list) were unaffected throughout.
   4. The live units were hand-written files, not home-manager output (the note's census said hand-installed; confirming) — the ported module is their first declarative form.
   5. The sandbox install needed a static full build (fleet-static, added here) — the step-1 variant set had static linkage only on the connect profile.
6. dezren39

   closed this as completedon Nov 4, 2026