fleet(secrets-broker): a partial vault refresh silently evicts cached secrets — one failed bws get becomes a 404 and a 'successful' refresh #5504

Problem

pkgs/fleet/crates/fleet-queue/src/secrets_broker.rs — do_refresh silently drops every secret whose individual bws secret get fails, then installs the shrunken map as the whole cache and stamps last_refresh as if the refresh had fully succeeded. One transient per-secret failure therefore evicts a live secret from the broker until the next refresh, and consumers are told 404 {"error":"no such secret"} — the answer for a secret that does not exist — for a secret that does.

This contradicts the broker's own stated refresh contract in run() (secrets_broker.rs, background-loop comment): "stale-while-revalidate (the cache keeps serving during a refresh, and a failed refresh keeps the old values)". That contract holds only when the whole refresh fails early (list/token stage). It does not hold for the per-item stage:

• The 8 worker threads fetch each value with nested if let Ok(...) (bws run, JSON parse, value field) — every failure shape is swallowed with no count, no log, and no error propagation.
• do_refresh then unconditionally replaces state.secrets with the partial map, sets last_refresh = now, and returns Ok(n) with the reduced count; the refresh loop logs it as an ordinary success.

Verified this run (behavioral probe of the real binary on the laptop)

Probe dir ~/git/fleet/probe-secrets-refresh (laptop): the shipped debug fleet binary at main 31a00fb run as fleet serve secrets-broker on 127.0.0.1:14899 with a fake bws on PATH serving two secrets (alpha.name, beta.name), a fake HOME holding only the broker token file, and TTL 900s.

1. Initial load: GET /health → {"ok":true,"secrets":2,"last_refresh_unix":…}; GET /secret/beta.name (authed) → 200 {"key":"beta.name","value":"value-beta"}.
2. The fake bws was flipped so bws secret get for beta's id exits 1 (alpha still succeeds), then POST /refresh → 202 {"ok":true,"refresh":"started"}.
3. After that refresh: GET /health → {"ok":true,"secrets":1,"last_refresh_unix":<advanced>}; GET /secret/beta.name → 404 {"error":"no such secret"}; GET /secret/alpha.name → still 200.

So a single transient bws failure turned an existing secret into a 404, with the health endpoint presenting the partial load as a successful refresh.

Why it matters

The secrets broker is the fleet's one secrets-by-name cache (code#5362): the queue broker fetches its Supabase credentials through it, and sandbox consumers resolve vault secrets through it. A flaky bws call, a vault hiccup on one item, or one malformed record currently degrades the whole cache silently — downstream failures surface far away as "no such secret" / credential errors, with nothing in the broker's log or health output saying the refresh was partial. Worse, if every get fails while list succeeds, the cache is emptied entirely and last_refresh still advances, so /secret/<name> returns 404 instead of the 503 "not loaded yet" shape a never-loaded cache gives.

Acceptance criteria

• A refresh in which any listed secret's get fails (non-zero exit, unparseable output, or a missing/non-string value) does not shrink the cache: either the refresh fails as a whole (old map and old last_refresh kept, failure logged — matching the documented stale-while-revalidate contract) or the failed names explicitly retain their previous values and the partial state is surfaced in the log/health output. Silently installing a partial map as a full success is the bug.
• The chosen behavior is pinned by a test or a scripted fake-bws probe (the probe recipe above reproduces the current behavior end-to-end).
• Controls unchanged: a fully successful refresh still swaps in new values and advances last_refresh; a secret genuinely removed from the vault listing still disappears on the next successful refresh.
• cargo test -p fleet-queue --all-features green (laptop or CI).

Found by new-issue-creator-code-b (Lane B) during the 2026-11-04 run; verified against the real binary and the source at main 31a00fb (do_refresh / run in pkgs/fleet/crates/fleet-queue/src/secrets_broker.rs).