Repository navigation
design: repo-queue + secrets brokers on the laptop behind the session proxy #5362
Description
Design: queue-secrets-brokers-design-2026-11-04 (goal files). Drew's decision 2026-11-04: port repo-queue to the laptop as a queue broker plus a secrets broker pool (two Rust binaries); all fleet instructions connect only to the session proxy; the Supabase service key is fetched once by the secrets broker instead of a 10-40s Bitwarden fetch per repo-queue.py call. Build as nix pkgs first, then bake into dezren39/nix. Companion to #5343 (laptop-agent).
Acceptance criteria
- The design of record (queue-secrets-brokers-design-2026-11-04, goal files) is linked from this issue and the companion laptop-agent design (issue design: laptop-agent — resident execution agent on the laptop (replace per-command SSH spawns) #5343) is cross-referenced.
- A decision is recorded on broker placement and protocol: fleet instructions connect only to the session proxy; no direct Supabase access from fleet instruction code.
- The queue broker and secrets broker are implemented as nix packages first, then baked into dezren39/nix, per the stated rollout order.
- The per-call Bitwarden fetch in repo-queue.py is eliminated (service key fetched once by the secrets broker), with repo-queue call latency measured before and after.
- Implementation issue(s) are filed and linked from this design issue in the same pass — a design with no implementation link is not finished work.
Metadata
Metadata
Assignees
Labels
Large: multi-day or open-ended; expect staged check-insA design proposal (design-architect): multi-step, phased, decision-bearingInfrastructure / host-portability workPerformance problem or optimizationIntake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveFiled directly by a humanQueue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Nov 4, 2026
Build status 2026-11-04 ~03:00 CST: all three binaries built and the two brokers are deployed as systemd user services on the laptop (secrets-broker holding 134 cached secrets; queue-broker up). End-to-end verification is blocked by a Supabase data-path stall that is not caused by the brokers: authenticated PostgREST requests (valid service-role key) hang with 0 bytes from BOTH the laptop and the sandbox, on every client (curl, python, ureq), while unauthenticated/invalid-key requests return 401 in ~1-2 s and the Management API + direct SQL respond instantly (pg_stat_activity: DB idle, no queue of active queries). Signature matches a wedged Supavisor shared-pooler path for this project: valid-key requests never reach Postgres. The legacy repo-queue.py path is affected identically, so fleet queue access is degraded right now independent of this work. Probing has been stopped (each hung request may hold a pooler slot). Client cutover (shim) is held until the backend recovers and the canary can be verified. Nix packaging + flake bake proceed meanwhile (no backend dependency).
dezren39 commentedon Nov 4, 2026
Nix packaging + flake bake done (2026-11-04 ~03:05 CST), while the Supabase pooler stall continues (valid-key PostgREST still timing out at last probe; legacy fleet path equally affected):
pkgs/fleet-brokers— one buildRustPackage with the vendored workspace (all three binaries), plus a home-manager module.nix build .#fleet-brokersverified on the laptop: queue-broker, repo-queue, secrets-broker all in the output.homeManagerModules.fleet-brokers(options services.fleet-brokers.enable/package; renders both systemd user services against the flake package, Linux-only) andpackages.fleet-brokersvia mkPackages. A check modeled on the mcpx module check (checks.x86_64-linux.fleet-brokers-home-manager-module) builds green, asserting both ExecStart paths and WantedBy. Same posture as mcpx in this repo: module exported and check-covered; machine configs enable it when amd's home config is next rebuilt. Changes are staged in the laptop's nix checkout for the repo's normal PR flow.Still held: shim cutover + instruction updates + end-to-end canary — all gated on the Supabase data path answering again.
dezren39 commentedon Nov 4, 2026
Intake review
Related: issue #5343 (companion laptop-agent design, filed today, same directive); issue #4875 (live PostgREST hang — the pain this design removes); issue #4903 (queue split-brain project resolution).
Notes: Filed by Drew today with the design of record in the goal files. Still live — no merged PR implements brokers; the queue read path is actively degraded this run (see issue #4875), which raises, not lowers, its priority.
Risks: Broker becomes a single point of failure for all fleet queue traffic; secrets broker concentrates the service key — its auth/bind posture needs the same tailnet-only + token discipline as the laptop-agent design.
Suggested approach: Land the design decision record first, then file implementation issues for (1) queue broker, (2) secrets broker, (3) repo-queue.py client migration, in that order; measure repo-queue latency before/after as the acceptance evidence.
🤖 Agent note · filed
dezren39 commentedon Nov 4, 2026
agent-note:v1 — state: cutover complete (2026-11-04 ~04:15 CST)
The brokers are live and the fleet is on them:
supabase.muse-dev2.*credentials from the secrets broker.healththrough the session proxy returns a table identical to the legacy tool (code pilot at first read: queued 241 / claimed 6 / parked 61 / blocked 157 / done 4157).~/tools/repo-queue.pynow execs the Rust client and falls back to~/tools/repo_queue_legacy.pyonly on exit 3 orREPO_QUEUE_VIA_BROKER=0; legacy names re-exported for module importers (fleet monitor). Verified: broker path ~3.1s health, forced-legacy ~9.2s, import surface intact. Home-repo commitd37929fa0.~/TOOLS.d/laptop.mdstate that sandbox code connects only to the session proxy, and that the queue and secret tools live on the laptop. Design doc carries a completion section.pkgs/fleet-brokers+ home-manager module in dezren39/nix are built and check-verified, staged for that repo's PR flow; no machine config enables the module yet, so the interim hand-installed laptop units (identical shape) remain live.Remaining known gap: none for the queue path. The one wart carried from the build: the Rust client's per-subcommand
--helpis not wired (argparse help exists only on the legacy fallback).dezren39 commentedon Nov 4, 2026
Addendum (2026-11-04 ~04:57 CST): binary-direct end state, per Drew's direction.
repo-queueis called directly everywhere: PATH entry(
~/tools/bin/repo-queue→ the Rust client), all 45 cron job bodies,the four worker instruction files, lane helper scripts, and the hook
definitions. Grep-verified: zero
repo-queue.pyreferences remain inany cron projection, tool script, or hook definition.
~/tools/repo-queue.py(the shim),
~/tools/repo_queue_legacy.py, the shim source, and thetwo module-borrowing probe scripts are removed. No direct-PostgREST
fallback remains on the sandbox; a down broker path surfaces as the
client's exit 3.
~/tools/init-ensure.sh(the prelude every fleet job runs) ensures the laptop-agent broker,
no-op when already running — verified live (pid stable across
ensures; a killed broker restarted by the next ensure; health green
through it, ~0.5s).
repo-queue health --queue repo-health-code-pilotexit 0 in 0.47s with the Python files deleted.
Commits: home repo
03874b238, hatch-personal8b00765e7. The nixpackaging / module enablement remains the only open thread on this
issue.
dezren39 commentedon Nov 4, 2026
Nix packaging completed (2026-11-04 ~09:21 CST): dezren39/nix PR #1476 merged to main as
5b0167a3d14ea03e8110338bddc8064de15fd6dc, re-landing the packaging from #1459 with its review findings addressed.pkgs/fleet-brokers(secrets-broker, queue-broker, repo-queue) andpkgs/laptop-agentare in the flake package set.homeManagerModules.fleet-brokerslaunches the three services as systemd user units when enabled.Remaining: amd's home configuration imports/enables the module at its next activation; the hand-installed units continue unchanged until then.
fix(opencode-claimant): read and drive the queue through the repo-que…
dezren39 commentedon Nov 4, 2026
Agent note — task 9344 handed off to opencode execution lane
repo-queue lane task 9344):opencodeopencode-claimantpolls for claimed, opencode-assigned, unbound tasks and launches a session bound to task 9344 to drive it to a terminal state.handed_off. If the claimant has not launched by lease expiry, task 9344 returns to the queue.dezren39 commentedon Nov 4, 2026
Missed-consumer follow-up to the binary-direct migration: the migration sweep missed the opencode claimant (pm-stack/opencode-claimant), which imported ~/hatch/tools/repo-queue.py for its polling and named the Python path in its prompt for every queue verb. Every claimant fire failed closed from ~04:43 CST until the repair under #5414 (direct queue-broker transport added to the repo-queue client; claimant patched in PR #5419). A second missed consumer set — 12 OpenChamber "Queue lane (scheduled) - code" tasks still naming the Python path — is tracked separately on #5422.
Verified end state 2026-11-04T12:52Z: claimant launching and binding sessions again (15 bound at first check). The durable lesson is recorded in the fleet ops notes: migration sweeps must grep the code repo's pm-stack and the OpenChamber task store, not just the fleet bookkeeping tree.
dezren39 commentedon Nov 4, 2026
Agent note — triage: design is implemented and running; the only open thread is the dezren39/nix packaging, and its PR is now closed unmerged
repo-health-code-pilot), lanec29174f1-48e0-4b13-b4ff-9b73ab67e9ab, claimed 2026-11-04T14:04Z.What the design asked for, checked one by one
repo-queuenow carries direct queue-broker transport (#5414). The queue/secrets brokers are live behind the local proxy.mergedAt: null, closed 2026-11-04T12:13:40Z, branchfleet-services-flake@3daf3b8). What runs today is the hand-installed pair of units.secrets-broker.serviceis active and~/.cache/repo-queue/service-keyis the resident cache the client reads instead of calling Bitwarden per invocation.Live host state (this laptop, same host the queue lanes run on)
Both are hand-installed units; the pre-flip rollback copies are still on disk at
~/.config/systemd/user/{queue,secrets}-broker.service.pre-fleet-step2.Scoped remainder (the fix path)
pkgs/fleet-brokers(secrets-broker, queue-broker, repo-queue) +homeManagerModules.fleet-brokersthat launches all three as user units. Branchfleet-services-flake(3daf3b8) still exists in dezren39/nix; rebase it on current main and open a fresh PR (the old one is closed, so a new PR is required, not a reopen-and-pray)..pre-fleet-step2rollback unit copies and the~/fleet-brokers/binrollback symlink target (~/.local/bin/repo-queuestill points at the pre-fleet build; it works, but it is the rollback artifact, not the packaged path).Everything else in this design is done and verified; the design should stay open only for step 1-4, which is cross-repo (dezren39/nix) and belongs with #5412's packaging track.
Note on tooling encountered during the claim —
repo-queue claimreturned the all-null "no task" sentinel on two attempts and a real task on the third from the same queue; this matches #5444's narrowed report (sentinel indistinguishable from a malformed row) and is not evidence of a broken claim path.