Skip to content

fix(codex-proxy): token sync to OpenCode store never runs — JSON.stringify shell quoting kills the Python child, and the error is swallowed #2533

Closed

Description

@dezren39

Observed 2026-10-29 on amd (laptop) by Muse, while verifying the Codex pool after a fresh codex login.

Symptom

After codex login refreshed ~/.codex/auth.json (2026-10-30T02:51:23Z), the laptop codex-proxy's built-in 5-minute syncCodexTokensToOpenCode did NOT update the OpenCode credential store. The active openai credential row still held a stale token (matching neither the old nor the fresh Codex token) until it was synced by hand. The proxy process runs the current script (since Oct 28), syncs at startup and every 5 minutes, and showed no error anywhere — the failure is invisible by design.

Root cause (diagnosed + reproduced 2026-10-29)

bin/codex-proxy.mjs, syncCodexTokensToOpenCode:

execSync(`python3 -c ${JSON.stringify(script)}`, { stdio: 'ignore' });
  1. The sync can never run. JSON.stringify(script) encodes the Python script's newlines as literal two-character \n sequences. The shell, inside double quotes, does not turn \n into a newline, so python3 -c receives the whole script as one physical line full of literal \n. Python dies instantly with SyntaxError: unexpected character after line continuation character. Reproduced in isolation against a scratch sqlite DB with the exact construction from the proxy: same SyntaxError, DB unchanged.
  2. The failure is swallowed twice. stdio: 'ignore' discards the child's stderr, and the surrounding catch (err) { // Non-fatal background sync } discards the throw. A failed sync is indistinguishable from a successful one — no log line either way.
  3. Latent overreach (would bite once the sync works): the embedded SQL selects WHERE integration_id="openai" with no active filter and updates every oauth row it finds. On the live DB that is TWO rows: the active base row (label='OpenAI', active=1) and an inactive second row (label='OpenAI 2', active=0, untouched since Oct 5). The sync must only ever touch the active base openai row — never the inactive row, never openai.alternate.

Fix

  • Run Python via execFileSync('python3', ['-c', script]) (argv, no shell) so the script arrives verbatim; pass the tokens + DB path via environment variables instead of interpolating them into the Python source.
  • Restrict the update to integration_id='openai' AND active=1 (plus the existing type='oauth' check in the row value).
  • Make outcomes visible: log success ([codex-proxy] codex-token-sync: updated N credential row(s)) and log failures to stderr with a clear codex-token-sync FAILED: marker including the child's stderr. Never swallow silently again.
  • Regression tests in bin/codex-proxy.test.mjs using the existing fake-HOME harness: seed ~/.codex/auth.json + a scratch opencode.db, start the proxy, assert only the active base openai row was updated (inactive + openai.alternate rows untouched) and time_updated advanced.

Related

Acceptance

  • Sync verified live on amd: after a proxy restart, the active openai row's time_updated advances and its access/refresh hashes match ~/.codex/auth.json.
  • GET /api/quota/codex still returns ok:true after the roll.
  • A deliberately broken sync (e.g. unreadable DB) produces a visible codex-token-sync FAILED log line in the proxy journal.

Activity

  1. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    source/sweepFiled or worked by an issue-sweep job
    priority/highImportant and time-sensitive
    effort/sSmall: expected under 2 hours of work
    app/codex-proxybin/codex-proxy.mjs fleet proxy
    on Oct 30, 2026
  2. dezren39 commented on Oct 30, 2026

    @dezren39
    MemberAuthor

    Diagnosis update (2026-10-29, fixing agent): a SECOND independent failure layer was found while preparing the deploy, and the fix approach changed accordingly.

    Layer 2: there is no python3 on the deployed proxy's PATH at all. The serving process (fps-codex-proxy-18744, PID 3133249) runs with PATH=/run/current-system/sw/bin:/home/user/.local/bin:/usr/bin:/bin. Verified live: python3 exists in none of those (/run/current-system/sw/bin/python3, ~/.local/bin/python3, /usr/bin/python3, /bin/python3 all absent — on NixOS it lives at /etc/profiles/per-user/user/bin/python3, which the service PATH does not include). So even with the quoting bug fixed, the Python child could never have spawned in the deployed environment.

    Fix as implemented in PR #2538: the sync no longer uses a child process at all — it runs in-process via node:sqlite (DatabaseSync, lazily createRequired so a missing module degrades to a logged failure, never a dead proxy). This removes the quoting class, the PATH/interpreter dependency, and all token interpolation. Scope (active base openai row only) and visibility (success line + codex-token-sync FAILED: stderr marker) are as specified in the issue. Both new regression tests pass on the laptop (~0.4s).

    CI note: repo-wide nix ci runner stall (all runs queued since ~2026-10-30T03:07Z) — the required check's script was run verbatim against the branch on the laptop and passes; PR is in the merge queue. Live deploy + sync verification results will be noted here when done.

  3. dezren39 commented on Oct 30, 2026

    @dezren39
    MemberAuthor

    FIX DEPLOYED + VERIFIED LIVE (2026-10-29 ~22:47–22:53 CDT). PR #2538 (head 82d94fa) is in the merge queue; repo-wide GitHub runner stall (every nix ci run queued since ~03:07Z) is the only thing between the PR and merge — the required check's script was run verbatim against the branch on the laptop and passes. Because the fix-it-now order does not wait on stalled runners, the deploy was landed from the pushed branch content and is hash-verified identical (primary bin/codex-proxy.mjs sha256 dad5f798… == branch blob). When the queue merges, the primary checkout becomes a no-op.

    Deploy actions:

    Verified sync cycles (the thing that never worked before):

    • Startup cycle, journal 22:47:11 CDT: [codex-proxy] codex-token-sync: updated 1 credential row(s).
    • Active row cred_10b5fd438001thsQzMXLQMc9oD: time_updated 1793328924205 (2026-10-30T02:55:24Z, the earlier hand sync) -> 1793332031557 (03:47:11.557Z); access sha256/12 97d43ababf95 and refresh 307b668ff3db — both exactly match ~/.codex/auth.json; expires = time_updated + 30d.
    • Periodic cycle: second log line 22:52:11 CDT (the 5-minute interval), time_updated advanced again to 1793332331565 (03:52:11.565Z).
    • Inactive row OpenAI 2 (active=0) and openai.alternate row: timestamps and hashes byte-identical before/after — untouched, as required.
    • Failure visibility, live journal proof: a throwaway systemd-run probe unit with a corrupt scratch DB logged [codex-proxy] codex-token-sync FAILED: file is not a database at 22:53:14 CDT (probe unit stopped and removed; live data untouched). The corrupt-DB regression test asserts the same marker.
    • Final read: GET /api/quota/codex active entry ok:true, weekly usedPercent 34, account drewrypope@gmail.com (baseline before restart was ok:true at 33%). Proxy /health ok on 18744.

    Remaining: merge of PR #2538 when runners recover (auto-merge/queue already armed; it will auto-close this issue via Fixes #2533).

  4. added and removed on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/codex-proxybin/codex-proxy.mjs fleet proxyeffort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)priority/highImportant and time-sensitivesource/sweepFiled or worked by an issue-sweep jobstatus/in-progressQueue worker lifecycle state

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions