Skip to content

providers: stale wellknown API-key wiring in the shared service sends an empty key after auth login/logout #54645

Open

Description

@brenwhyte

Summary

With the v2 shared background service, changing which well-known endpoints are authenticated (opencode auth login / opencode auth logout) does not update the provider's API-key substitution wiring. Requests keep resolving {env:VAR} for a previously-authenticated endpoint whose credential no longer exists, so an empty API key is sent and the upstream provider rejects the request with 401. v1.19.0 and v2 --standalone are unaffected.

Environment

  • opencode version: 2.0.26 (also tested: 1.19.0)
  • OS: Darwin 25.6.0 (macOS, arm64)
  • Terminal: xterm-256color, TERM_PROGRAM=vscode, truecolor (observed from the reporting shell)
  • Shell: /bin/zsh
  • Install/channel: latest (local=false)
  • Active plugins: oh-my-opencode-slim@3.0.3

Reproduction

Setup: two well-known endpoints, A and B. Each serves a manifest declaring a distinct auth.env name (VAR_A / VAR_B) and a remote config that sets the same provider's auth header to a different env name:

  • A's remote config: provider.<P>.options.headers["x-api-key"] = "{env:VAR_A}"
  • B's remote config: same header = "{env:VAR_B}"

Each endpoint's auth.command returns a different API key, and each key is valid only for that endpoint's backend.

Steps (shared background service — normal opencode usage, not --standalone):

  1. opencode auth login https://endpoint-a/...
  2. opencode run -m <P>/<model> "..." -> works
  3. opencode auth login https://endpoint-b/... (both A and B authenticated)
  4. opencode run ... -> works
  5. opencode auth logout https://endpoint-b/... (A authenticated again)
  6. opencode run ... -> fails: provider 401 Invalid API Key — even though A is authenticated and A's credential is present
  7. opencode auth logout https://endpoint-a/... then opencode auth login https://endpoint-a/... -> works again (the credential matches the stale wiring)
  8. opencode service restart while A is authenticated -> still fails (the stale state survives a restart)

Control experiments (same sequences):

  • v1.19.0: all steps pass.
  • v2.0.26 with --standalone: all steps pass.
  • v2.0.26 logged into both endpoints simultaneously (never logging out): works reliably.

Expected Behavior

After step 5, requests should use endpoint A's credential (A is the authenticated endpoint) and succeed. The provider's env-name wiring should follow the set of currently authenticated well-known endpoints.

Actual Behavior

Step 6 sends an empty/invalid API key and the provider returns 401 (Invalid API Key, provider error code 03 in this deployment's gateway). The auth state change itself reports success; the problem only appears when a request is made.

Additional Context

Server log excerpt (names genericized):

credential removed  integrationID=https://endpoint-b/...
event.type=config.updated
event.type=provider.updated
...
Failed to drain Session  cause="AI.Error: ... Error: Invalid API Key. Error Code: 03"

Notes:

  • wellknown:sources persists the list of endpoints ever seen, and cached well-known configs remain after logout. This appears to be why a service restart does not clear the stale wiring — a fresh boot can re-apply the stale env-name mapping from the cached configs.
  • The provider's apiKey is set to a placeholder ("dummy") with the real key supplied only by the header substitution, so an unresolved {env:...} yields an invalid key rather than a clear missing-auth error. That may be why the failure surfaces as a provider 401 rather than an auth error.
  • Frequency: deterministic once both endpoints have been authenticated on the same service; in day-to-day use it looks intermittent because it only breaks while the wiring's env name has no matching credential (i.e. after logging out of that endpoint).
  • Workarounds: stay logged into both endpoints; use --standalone; or use v1.x.
  • Internal URLs, endpoint names, and credentials have been genericized.

Activity

  1. opencode-agent commented on Oct 11, 2026

    @opencode-agent
    Contributor

    Thanks for the detailed report. I reproduced this on 2.0.26, and it still happens on the latest dev (707de58).

    I used a local mock with two well-known origins, A (VAR_A) and B (VAR_B). Each remote config sets the same provider's x-api-key header to {env:VAR_X}, with apiKey: "dummy". A mock gateway logs the key it receives:

    1. auth login A, then run → sends A's key, works
    2. auth login B, then run → sends B's key, works
    3. auth logout B, then run → sends x-api-key: "" and gets Invalid API Key. Error Code: 03, even though auth list shows A's credential is still there
    4. After restarting the service, run still sends an empty key

    The script below reproduces it end to end.

    From reading the code (not tested yet), the likely cause is this: logout removes only the credential, not the well-known source or its cached remote config. That cached config is still applied with no credential, so {env:VAR_B} resolves to an empty string and overrides A's header. That would also explain why a restart doesn't clear it.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # Reproduces anomalyco/opencode#54645. Mock: well-known origins on :4101/:4102, gateway on :4100.
    # OC=opencode (default, uses the shared background service), or OC=./oc-dev.sh MANUAL=1 to run a
    # long-lived `opencode serve` from source and connect with --server (the auto-spawned service fails from source).
    # Self-contained: writes and starts the mock servers (needs bun).
    M=$(mktemp -d /tmp/opencode/mock.XXXX)
    cat > "$M/mock.ts" <<'MOCK'
    // Two well-known origins (A on 4101, B on 4102) and a mock OpenAI-compatible gateway on 4100.
    const providerCfg = (env: string) => ({
      provider: {
        gw: {
          npm: "@ai-sdk/openai-compatible",
          name: "Gateway",
          options: { baseURL: "http://127.0.0.1:4100/v1", apiKey: "dummy", headers: { "x-api-key": `{env:${env}}` } },
          models: { m: { name: "m" } },
        },
      },
    })
    for (const [port, env, key] of [[4101, "VAR_A", "key-a"], [4102, "VAR_B", "key-b"]] as const) {
      Bun.serve({
        port,
        fetch(req) {
          const u = new URL(req.url)
          if (u.pathname === "/.well-known/opencode")
            return Response.json({
              auth: { command: ["sh", "-c", `echo ${key}`], env },
              remote_config: { url: `http://127.0.0.1:${port}/config`, headers: { authorization: `Bearer {env:${env}}` } },
            })
          if (u.pathname === "/config") return Response.json(providerCfg(env))
          return new Response("nf", { status: 404 })
        },
      })
    }
    Bun.serve({
      port: 4100,
      async fetch(req) {
        const k = req.headers.get("x-api-key")
        console.log(`GATEWAY ${req.method} ${new URL(req.url).pathname} x-api-key=${JSON.stringify(k)}`)
        if (k !== "key-a" && k !== "key-b") return Response.json({ error: { message: "Invalid API Key. Error Code: 03" } }, { status: 401 })
        const chunk = (d: any) => `data: ${JSON.stringify(d)}\n\n`
        const base = { id: "x", object: "chat.completion.chunk", created: 0, model: "m" }
        return new Response(
          chunk({ ...base, choices: [{ index: 0, delta: { role: "assistant", content: "OK" } }] }) +
            chunk({ ...base, choices: [{ index: 0, delta: {}, finish_reason: "stop" }] }) + "data: [DONE]\n\n",
          { headers: { "content-type": "text/event-stream" } },
        )
      },
    })
    console.log("mock up")
    MOCK
    bun "$M/mock.ts" > "$M/mock.log" 2>&1 & MOCKPID=$!; sleep 1
    trap 'kill $MOCKPID 2>/dev/null; echo "--- gateway log"; grep GATEWAY "$M/mock.log"' EXIT
    OC=${OC:-opencode}
    export OPENCODE_PASSWORD=${OPENCODE_PASSWORD:-repro-pass}
    export HOME=$(mktemp -d /tmp/opencode/home.XXXX) XDG_CONFIG_HOME= XDG_DATA_HOME= XDG_STATE_HOME= XDG_CACHE_HOME=
    P=$(mktemp -d /tmp/opencode/proj.XXXX); cd "$P"; git init -q
    A=http://127.0.0.1:4101 B=http://127.0.0.1:4102
    S=()
    start(){ [ -n "$MANUAL" ] || return 0; $OC serve --port 4199 >/tmp/opencode/serve.log 2>&1 & SP=$!; sleep 6; }
    stop(){ if [ -n "$MANUAL" ]; then kill $SP 2>/dev/null; wait $SP 2>/dev/null; else timeout 30 $OC service stop >/dev/null 2>&1; pkill -f "[s]erve --service"; fi; }
    [ -n "$MANUAL" ] && S=(--server http://127.0.0.1:4199)
    step(){ echo "=== $*"; }
    run(){ timeout 90 $OC run "${S[@]}" -m gw/m "say hi" 2>&1 | tail -2; }
    stop; start
    step "login A"; timeout 60 $OC auth login "${S[@]}" $A 2>&1 | tail -1
    step "run (A)"; run
    step "login B"; timeout 60 $OC auth login "${S[@]}" $B 2>&1 | tail -1
    step "run (A+B)"; run
    step "logout B"
    CB=$($OC auth list "${S[@]}" --format json | bun -e 'const j=JSON.parse(await Bun.stdin.text());console.log(j.find(i=>i.id==="http://127.0.0.1:4102").connections[0].id)')
    timeout 60 $OC auth logout "${S[@]}" $B "$CB" 2>&1 | tail -1
    echo "credentials left: $($OC auth list "${S[@]}" --format json | grep -c '"credential"')"
    step "run (A only) -> expected OK"; run
    step "restart server/service"; stop; start
    step "run after restart (A only)"; run
    stop

    Run it: bash repro.sh

  2. nosmile99 commented on Oct 12, 2026

    @nosmile99

    I reproduced this and intend to work on it — I have a fix with a regression test ready and will open a PR against dev shortly.

    Root cause (verified empirically): in packages/core/src/config.ts (loadWellknownEntry), when the credential for a well-known endpoint is gone (logged out), the code falls through to wellknown.cached(entry.origin) and substitutes the cached remote config with env: undefined. ConfigVariable.substitute then resolves {env:VAR_B} to "", so the logged-out endpoint still emits a provider document whose empty x-api-key header overlays the still-authenticated endpoint's wiring (later document wins in mergeHeaders) — hence the 401. The stale state survives restarts because the cached config lives in KV until the source is removed.

    The fix: when the manifest declares auth but no usable key credential exists, the entry contributes no documents, so the provider wiring follows the set of currently authenticated endpoints. A failed refresh with a credential present still keeps the last resolved config.

  3. nosmile99 commented on Oct 12, 2026

    @nosmile99

    PR is up: #54687 (against dev, per the contribution gates). Root cause and verification are in the PR body; happy to adjust if the approach needs rework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions