remote-control 409 stale enrollment needs supported takeover or repair path #28862

Summary

I hit what looks like the stale remote-control enrollment / 409 conflict failure mode. I was able to recover locally, but the recovery path points to a deeper issue: Codex appears to need a supported way for a new app-server process to take over from a stale or ghost remote-control connection.

This looks related to several existing remote-control stale-state reports, but the key difference here is that clearing remote_control_enrollments alone was not sufficient; the backend continued returning 409 until the local host identity was rotated.

Environment

• Codex CLI entrypoint: codex-cli 0.136.0
• Managed remote-control app-server: 0.141.0
• Latest npm package checked at report time: @openai/codex 0.141.0
• Install note from codex doctor: the CLI entrypoint and npm update target are from different npm roots, but the managed remote-control app-server was already 0.141.0.
• Platform: Linux 6.17.0-1016-gcp x86_64 x86_64
• OS: Ubuntu 24.04 GCP Linux VM
• Terminal/multiplexer: SSH + tmux 3.4
• Auth mode: ChatGPT auth
• Remote surface: ChatGPT mobile Codex remote-control view

Relevant current health after workaround:

{
  "mode": "daemon",
  "status": "connected",
  "daemon": {
    "status": "alreadyRunning",
    "backend": "pid",
    "cliVersion": "0.136.0",
    "appServerVersion": "0.141.0"
  }
}

What issue are you seeing?

Codex remote control can report that remote control is enabled for a host, but the connection is errored:

Remote control is enabled on <host> but the connection is errored.
HTTP error: 409 Conflict
{"detail":"Remote app server already online"}

A related mobile symptom is that the remote host can appear green/reachable, but the mobile app and remote host appear to be attached to diverged environments or session lists. Both sides can still call/use Codex, but recent sessions are not reflected consistently.

What steps can reproduce the bug?

I do not have a minimal deterministic reproducer yet, but the observed path was:

1. Run Codex remote control on a long-lived Linux VM.
2. Accumulate a stale/ghost remote-control state where the backend believes an old app-server is still online.
3. Attempt to restart/reconnect remote control.
4. Observe 409 Conflict with {"detail":"Remote app server already online"}.
5. Stop remote control and kill lingering codex remote-control / codex app-server processes.
6. Clear remote_control_enrollments from ~/.codex/state_5.sqlite.
7. Attempt to start remote control again.
8. In this case, the backend still returned 409 until the local installation_id was rotated.

Expected behavior

Remote control should provide a supported recovery path when the backend believes an older app-server is still online but the local host is trying to reconnect.

At minimum, the CLI should distinguish this condition from a generic errored connection and guide the user through a safe repair flow. Ideally, the system should support authenticated takeover by the new app-server process without requiring manual SQLite edits or host identity rotation.

Actual behavior

The practical recovery required local state surgery:

1. Stop codex remote-control.
2. Kill lingering codex remote-control, codex app-server, and related remote-control processes.
3. Back up ~/.codex/state_5.sqlite and ~/.codex/logs_2.sqlite.
4. Clear only Codex remote-control enrollment state from remote_control_enrollments.
5. Start remote control again so Codex creates a fresh enrollment.

In my case, clearing remote_control_enrollments alone was not sufficient. The backend still returned 409 until the local installation_id was rotated. After that, codex remote-control start --json returned connected, the local enrollment table had one current row, and the managed app-server held the active websocket.

That suggests the backend can continue associating an old installation identity with an online app-server even after local enrollment state has been deleted.

Root-cause hypothesis

The current implementation appears to conflate three concepts that should be modeled separately:

stable host identity
  installation_id

registered remote host
  server_id
  environment_id

current live connection
  lease_id / connection_epoch

A persisted enrollment should identify the registered host/environment, but it should not itself be the authority for whether a particular app-server process is the current live connection.

The backend seems to reject a new connection with 409 when it believes an older app-server for the same host/enrollment is still online. If that older connection is a ghost, stale lease, or no longer user-visible, the CLI has no supported way to take over or repair the condition other than local state surgery and, in some cases, host identity rotation.

Proposed fix direction

Introduce backend-backed connection leases with fencing/takeover semantics.

Suggested invariant:

For a given authenticated host identity, exactly one app-server lease is current, and all older leases are explicitly superseded.

Suggested behavior:

1. Each app-server process generates a fresh lease_id or monotonic connection_epoch when remote control starts.
2. WebSocket connect or server refresh includes installation_id, server_id, and the current lease.
3. If the backend sees an existing online connection for the same authenticated host identity, it allows an authenticated takeover after a short grace period or with an explicit takeover=true path.
4. The backend marks the older lease superseded and closes or rejects traffic from it.
5. Mobile/remote clients route only to the newest active lease for the host/environment.
6. The CLI treats 409 Remote app server already online as a recoverable stale-lease condition, not a generic terminal connection error.

CLI-side recovery improvements

Even before or alongside backend lease work, the CLI could improve the user-facing recovery path:

• Detect 409 Remote app server already online specifically.
• Retry briefly in case the old websocket is genuinely shutting down.
• If still 409, attempt a bounded re-enroll/takeover flow if supported by the backend.
• If takeover is unavailable, surface a specific repair action instead of a generic errored connection state.
• Add a supported codex remote-control repair or codex remote-control reset command that:
  • stops the managed remote-control app-server,
  • verifies no ghost process remains,
  • backs up local Codex state,
  • clears only remote-control enrollment state,
  • optionally regenerates host identity as an explicit last-resort step.

Test cases worth covering

• Persisted enrollment exists, but backend returns 409 because an older app-server is believed online.
• codex remote-control stop reports success while a long-lived app-server process still exists.
• Local enrollment row is deleted but the same installation_id still receives 409.
• Mobile host list shows a reachable host while local session/environment state is stale or diverged.
• Revoked or stale remote-control client state is reported distinctly from stale app-server lease state.

Related reports

• Remote control can get stuck after idle or network changes with 409 Conflict until re-enrollment #24024 - closed, but contains the same 409 shape and local recovery details.
• Android Codex shows stale local project threads despite connected desktop until remote-control enrollment reset #24749 - connected mobile host with stale/diverged thread list until enrollment reset.
• codex remote-control reuses persisted enrollment with stale server_name, causing silent WebSocket disconnect #25241 - persisted enrollment reuse with stale host identity fields.
• Codex Desktop remote control remains connected but unusable after remote device revoked permission, and cannot be deleted or re-authorized #23865 - revoked/reauthorization stale-state recovery gaps.
• Codex Desktop remote-control keeps using revoked client when connecting Mac to Windows, no approval request appears #25532 - stale/revoked client reuse.
• Remote Control goes offline while desktop WebSocket to chatgpt.com remains active #24179 - remote-control offline state while desktop websocket remains active.

Contribution boundary

If maintainers agree with the diagnosis, I can help with the public CLI-side part: 409 detection, clearer recovery messaging, repair/reset command behavior, and tests. The lease/takeover model likely crosses backend infrastructure and may need to be implemented internally.