Repository navigation
fleet(secrets-broker): serves stale secret value long after BWS rotation — read-through cache never invalidates #6767
Description
Problem
The laptop secrets broker serves a STALE secret value long after the
secret is rotated in Bitwarden Secrets Manager. fleet secrets get
(broker-first) keeps returning the pre-rotation value, so a rotation
appears to succeed while every broker consumer keeps using the dead
credential.
Observed (2026-11-06, during the runtime-db-export Turso repoint, #5691)
- BWS secret
turso.muse-mirror-rw-tokenrotated viabws secret edit;
verified by directbws secret get: new value sha256 fingerprint
6320fec2(len 348). fleet secrets get turso.muse-mirror-rw-tokenreturned the OLD value
(fingerprinta0542dc2) at +20s after the edit, and STILL at +10 min:
the repointed loader's live probe at ~22:05 CST had its broker-supplied
token rejected by Turso (auth role not found) and only succeeded via
a direct-BWS fallback.- The old token was not merely stale but WRONG for the current DB
(minted against a superseded muse-mirror instance), so the failure
mode was a hard auth rejection, not a soft staleness.
Impact
Any consumer resolving rotated secrets through the broker silently uses
dead credentials after a rotation (DB tokens, API keys). Rotation
runbooks that verify via BWS directly will report success while broker
consumers keep failing.
Expected
- The broker reflects BWS edits within its documented TTL (bounded,
minutes at most), orfleet secrets getdocuments that it may serve
cached values and exposes the value's age/source. - Regression check: rotate a canary secret in BWS, assert
fleet secrets getreturns the new value within the TTL.
Related: #5860 (secrets-broker vault-refresh loop defects — TTL=0 spin).
Workaround in place: ~/tools/runtime-db-export/turso_client.py falls
back to a direct BWS read when the DB rejects the broker-supplied token.
Acceptance criteria
- After a secret is edited/rotated in Bitwarden Secrets Manager,
fleet secrets get <name>returns the NEW value within a bounded, documented TTL (minutes at most) — no indefinite staleness. - A regression check exists: rotate a canary secret in BWS, then assert the broker-served value matches the direct-BWS value within the TTL.
- If a cached value is ever served, its age/source is knowable (documented behavior or exposed metadata), so a rotation runbook can tell whether it is verifying the broker or BWS.
- The stale-value failure mode from the 2026-11-06 observation (broker token rejected by Turso at +10 min after rotation) no longer reproduces.
Activity
dezren39 commentedon Nov 7, 2026
Intake review
🤖 Agent note · filed
Still live? Yes. Searched merged PRs for a secrets-broker cache fix: PR #5506 (partial refresh keeps previous cache, issue #5504) and PR #6292 (read-through + empty values fail loudly, issue #6275) predate this report and do not address post-rotation staleness — this issue's evidence is a fresh 2026-11-06 observation (during issue #5691 work, ~22:05 CST) where the broker still served the pre-rotation value at +10 min and Turso rejected it.
Related: issue #5860 (OPEN — SECRETS_BROKER_TTL_S=0 vault-refresh spin; same broker cache/refresh subsystem, likely worth fixing in the same pass or at least not regressing), issue #5691 (the work during which this was observed).
Risks: silent credential staleness is worse than a loud failure — rotation runbooks verify via BWS directly and report success while broker consumers keep failing on dead credentials. Any fix must keep the broker-first path fast (it is on the hot path for fleet tooling) and must not reintroduce the #5860 refresh spin.
Suggested approach: make the read-through cache TTL-bounded and/or keyed to a BWS-side version signal so an edit invalidates promptly; expose value age/source if a cached value is served; add the canary rotation regression check from the acceptance criteria. The direct-BWS fallback already in
~/tools/runtime-db-export/turso_client.pyis a workaround for one consumer, not the fix.developing-today commentedon Nov 8, 2026
agent-note:v1
state: triaged — CONFIRMED, mechanism pinned at current main, fix enqueued
Agent: repo queue worker lane code-r, task 11261 (triage of issue #6767)
Queue: repo-health-code-pilot
Revalidation (fresh, 2026-11-07)
Issue #6767 is OPEN, priority/high, no hard-stop labels, no linked PR. Read and this note via the developing.today App route (user-token GraphQL limit exceeded this shift). Source verified at current main
ab35bef4cae2babea2f9835b66293b33bacbf88d(2026-11-07 19:03 CST), filepkgs/fleet/crates/fleet-queue/src/secrets_broker.rs.Verdict: CONFIRMED — and the mechanism is more specific than "the cache never invalidates"
A TTL refresh exists, but three properties combine to produce the observed indefinite staleness:
GET /secret/<name>:Some(v) => (200, {"key", "value"})— a hit returns the cached value unconditionally, and the response carries no age or source metadata (only/healthexposeslast_refresh_unix). Nothing on the read path enforces or documents a staleness bound, so acceptance item 3 (age/source knowable) is unmet by construction.do_refreshcounts per-item failures and, iffailed > 0, returns Err and keeps the ENTIRE previous cache ("{failed} of {total} secret fetches failed; keeping the previous cache" — the fleet(secrets-broker): a partial vault refresh silently evicts cached secrets — one failed bws get becomes a 404 and a 'successful' refresh #5504 anti-eviction contract). One persistently failing secret therefore pins every other secret's rotated value stale, refresh after refresh, indefinitely. This is the "never invalidates" path: it needs only one bad item in the vault.sleep(TTL)withSECRETS_BROKER_TTL_Sdefault 900, so worst-case staleness is TTL + full-vault refresh duration (>15 min), and the 2026-11-06 observation of a stale value at +10 min sits inside a single healthy window. Read-through (secrets broker: no read-through on cache miss — newly minted secrets invisible up to 15 min, client returned empty #6275) does not help: it fires only on cache MISSES (new names), never on a rotated existing name.POST /refreshexists as a manual escape hatch, butfleet secrets getnever triggers it and abws secret editrotation runbook has no reason to know it exists.Related: #5860 (SECRETS_BROKER_TTL_S=0 refresh spin) is the same file/subsystem — note
ttl_sparsing here accepts 0 with no floor. The fix pass should land the TTL floor with this work or record explicit ordering against #5860; do not regress it.Fix scope (enqueued as the follow-up task)
/secretresponses (orfleet secrets getoutput) expose value age/source so a rotation runbook can tell whether it verified the broker or BWS.This triage task (task 11261) completes as
triaged; issue #6767 stays OPEN for the fix.dezren39 commentedon Nov 8, 2026
Agent note — task 13703 triage re-confirmation (lane code-x)
dezren39 commentedon Nov 8, 2026
Agent note — task 13703 structured outcome (lane code-x)
The queue complete verb for task 13703 returned the verdict task 13703 is not claimed by repo-worker-code (the claim was superseded/released outside this lane between the verified checkpoint and the terminal verb; the backend itself answered — this is a verdict, not an outage). Per runbook §0b, the full outcome is recorded here durably:
No GitHub work was lost: the triage verdict note 6728667843 and this outcome note are on issue #6767, and the fix route (task 11460) is live in the queue.