Skip to content

Rotate the agent-auth Meta secret at the source (sops still encrypts the stale key) #861

New issue
New issue
Open
Open
Rotate the agent-auth Meta secret at the source (sops still encrypts the stale key)#861
Assignees
dezren39
Labels
app/agent-shell-envPackage/app: agent-shell-env (pkgs/ or apps/ in this repo)effort/sSmall: expected under 2 hours of workkind/securitySecurity issue or security hardeningmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedmode/onlineFully doable remotely: design, config, code, cloudpriority/highImportant and time-sensitivesecurity-adjacentTouches security-sensitive surface; extra care and reviewsource/humanFiled directly by a humanstatus/blockedQueue or user escalation stateuser/escalatedQueue or user escalation state

Description

@dezren39
dezren39
opened on Oct 25, 2026

Problem

On 2026-10-25 the laptop's OpenCode sessions failed on the meta provider with 401 invalid_api_key, even though ~/.local/share/opencode/auth.json held the correct funded key. Root cause: agent-shell-env (BASH_ENV env.sh, __ase_load META_API_KEY) loads the key from /run/secrets/agent-auth/muse-sdk-token (fallback ~/.config/muse/api-key), and the OpenChamber server process carried that env value, overriding auth.json. Both files held a stale key (sha256 prefix 7d4f5006).

Hot-fix applied 2026-10-25 (live, verified)

Both secret files were replaced with the funded key (Bitwarden + 1Password item meta.api-key-funded, sha256 prefix 6b1ea8ee) and openchamber.service was restarted; laptop sessions then answered on meta/muse-wonder-1.0, content-verified.

The gap

The sops source in this repo — nixos/agent-auth/ — still encrypts the OLD key. Any secrets regeneration or laptop reboot restores the stale /run/secrets/agent-auth/muse-sdk-token and the 401s return.

Fix

  • Re-encrypt the agent-auth secret with the current funded key from the vault (meta.api-key-funded).
  • After the next regeneration/reboot, verify the deployed file's sha256 prefix is 6b1ea8ee and that a laptop OpenCode session answers on meta/muse-wonder-1.0.
  • Fold into the laptop declarative-module cleanup so the imperative hot-fix doesn't drift again.

Key values are referenced by name/hash prefix only — never paste the key into this issue, a commit, or chat.

Acceptance criteria

  • The agent-auth secret in nixos/agent-auth/ (sops) is re-encrypted with the current funded key (vault item meta.api-key-funded, referenced by name only — the key value never appears in a commit, issue, or chat).
  • After a secrets regeneration or laptop reboot, the deployed /run/secrets/agent-auth/muse-sdk-token has sha256 prefix 6b1ea8ee (not the stale 7d4f5006).
  • A laptop OpenCode session on the meta provider (meta/muse-wonder-1.0) answers successfully after that regeneration/reboot, content-verified.
  • The imperative hot-fix is folded into the laptop declarative-module cleanup so the secret source cannot drift stale again.

Activity

  1. github-actions
    assigned
    dezren39
    on Oct 25, 2026
  2. dezren39
    added
    kind/securitySecurity issue or security hardening
    source/humanFiled directly by a human
    status/needs-triageNot yet triaged; needs a first look
    priority/highImportant and time-sensitive
    effort/sSmall: expected under 2 hours of work
    app/agent-shell-envPackage/app: agent-shell-env (pkgs/ or apps/ in this repo)
    security-adjacentTouches security-sensitive surface; extra care and review
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 26, 2026

  3. dezren39 commented on Oct 26, 2026

    @dezren39
    dezren39
    on Oct 26, 2026
    MemberAuthor

    Intake review

    Still live (verified this run): the hot-fix described in the issue was applied to the live secret files only; no merged PR since rotates the sops source in nixos/agent-auth/, so a regeneration/reboot still restores the stale key (sha256 prefix 7d4f5006) and the 401s return. Not closed. Acceptance criteria appended to the body this pass.

    Related: the 2026-10-25 laptop meta-provider 401 incident this issue was filed from; vault item meta.api-key-funded (referenced by name only).

    Notes / risks: priority/high and security-adjacent — this is secret handling. The rotation itself is a vault-access task for Drew/Muse with the vault open, not a sweep-automation task: never paste the key into an issue, commit, or chat, and verify by sha256 prefix only (6b1ea8ee good, 7d4f5006 stale). Effort is small (s) once vault access is in hand; the blocker is access, not work.

    Suggested approach: re-encrypt the agent-auth sops secret from the vault item, regenerate/deploy, then verify the deployed file's prefix and a live laptop session on meta/muse-wonder-1.0; fold the source into the declarative-module cleanup so the hot-fix cannot drift again.

    🤖 Fleet monitor intake enrichment — labels normalized, acceptance criteria appended, and meta/enriched applied this pass.

  4. dezren39
    added a commit that references this issue on Oct 26, 2026

    fix(nixos): re-mirror bws sops with rotated muse.sdk-token (funded Me…

    541f95e

  5. dezren39 commented on Oct 26, 2026

    @dezren39
    dezren39
    on Oct 26, 2026
    MemberAuthor

    Rotation at the source — done (2026-10-26, laptop batch). Keeping this open for the two residuals below.

    What was done

    • Bitwarden muse.sdk-token still held the stale key (verified by hash only: 7d4f5006). Updated it vault-to-vault from meta.api-key-funded; it now hashes to 6b1ea8ee. This was the true source: the sops mirror is generated from bws, so the stale value kept re-encrypting itself.
    • Regenerated secrets/sops/common/bws.yaml with lib/bws-to-sops.sh on the laptop (full mirror, 95 secrets; no existing keys dropped). Merged as PR fix(nixos): re-mirror bws sops with rotated muse.sdk-token (funded Meta key) #969 (merge commit d0709ae9; checks green: flake evaluation, treefmt, CodeRabbit). Decrypting muse.sdk-token from the merged file verifies at sha256 prefix 6b1ea8ee.
    • Confirmed the failure mode live this morning: the laptop rebooted and /run/secrets/agent-auth/muse-sdk-token was back to 7d4f5006, exactly as this issue predicted. Re-patched the live file to the funded key (verified 6b1ea8ee, owner/mode restored) and relaunched the OpenCode server under the corrected environment; the server process env now carries the funded key (hash-verified).

    Residuals

    1. System switch still pending. The laptop's running NixOS generation is from 2026-10-10 and still embeds the old sops file, so a reboot before the switch restores the stale key again. Once the laptop switches to a generation containing PR fix(nixos): re-mirror bws sops with rotated muse.sdk-token (funded Meta key) #969, the after-reboot criterion holds permanently. (The same pending switch also carries the sshd MaxStartups raise.)
    2. Meta sessions now fail on billing, not credentials. With the funded key in place, the 401 invalid_api_key is gone; a verification session on meta/muse-wonder-1.0 against the laptop server now returns 402 billing_not_configured ("Billing verification failed… your team needs to resolve a billing issue first", team_id 1073226932208501). That is Meta account state, resolvable only in the Meta billing console — flagged to Drew separately. The session-answers acceptance criterion is blocked on that, not on this rotation.
  6. dezren39
    added
    mode/onlineFully doable remotely: design, config, code, cloud
    work/assignedQueue worker lifecycle state
    and removed
    status/needs-triageNot yet triaged; needs a first look
    on Nov 1, 2026

  7. 7 remaining items

  8. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    Triage verdict against current main 2f62ce6 (no secret values read or decrypted — structure and git history only):

    Repo-side rotation is done. PR #969 (merge d0709ae, in main) rotated muse.sdk-token in secrets/sops/common/bws.yaml; PR #1320 (merged 2026-10-27) re-synced the mirror after the meta key renames; HEAD still carries muse.sdk-token plus both renamed meta.*.api-key entries. Consumers unchanged: nixos/agent-auth/default.nix:52 still maps agent-auth/muse-sdk-token to bws key muse.sdk-token, and agent-shell-env still loads META_API_KEY from that path. No follow-up fix packet enqueued — there is no remaining repo-side code change.

    What remains is not queue work (matches Drew's 2026-10-26 note, which deliberately kept this open): (1) laptop nixos-rebuild switch to a generation containing the rotated mirror, otherwise a reboot restores the stale key; (2) the Meta 402 billing_not_configured on the funded key, resolvable only in the Meta billing console. Both need Drew/laptop-operator action, not a worker packet.

    Queue labels resynced to queued; leaving open per Drew's standing call.

  9. dezren39
    added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026

  10. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    🤖 Agent note · picked up

  11. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    🤖 Agent note · triaged

    • Verification: PR fix(nixos): re-mirror bws sops with rotated muse.sdk-token (funded Meta key) #969 (merged 2026-10-26, merge d0709ae9) rotated muse.sdk-token in secrets/sops/common/bws.yaml; PR Re-sync bws sops mirror after meta key renames #1320 (merged 2026-10-27, merge 1bf2afb2) re-synced the mirror after the Meta key renames. No merged PR since has changed that path, and the issue itself records that the rerotation was hash-verified at the funded prefix.
    • Decision: No actionable follow-up packet. Remaining work is human-side: a NixOS generation switch on the laptop before reboot, and Meta billing-console action for the funded-key 402 billing_not_configured. Neither is executable as a repository code change.
    • Routing: Escalate to Drew for those two decisions; do not leave the issue as an ownerless repository work order.
  12. dezren39
    added
    user/escalatedQueue or user escalation state
    queue/queuedHas a queued task in the Supabase agent queue
    and removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 3, 2026

  13. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    🤖 Agent note · triaged

  14. dezren39
    added
    status/blockedQueue or user escalation state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/agent-shell-envPackage/app: agent-shell-env (pkgs/ or apps/ in this repo)effort/sSmall: expected under 2 hours of workkind/securitySecurity issue or security hardeningmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedmode/onlineFully doable remotely: design, config, code, cloudpriority/highImportant and time-sensitivesecurity-adjacentTouches security-sensitive surface; extra care and reviewsource/humanFiled directly by a humanstatus/blockedQueue or user escalation stateuser/escalatedQueue or user escalation state

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions