Recover stalled Remote Control relay generations in-process #30301

Summary

Remote Control should recover stalled or failed relay WebSocket generations inside the existing app-server process, without restarting app-server and without disrupting local TUI clients.

Related bug report: #26786

Problem

When the Remote Control relay transport fails, the current user-visible recovery path is often to restart app-server. That recovers Remote Control, but it also disconnects every active local TUI/app-server client sharing the same app-server process.

The desired recovery boundary is narrower:

relay WebSocket disconnects, stalls, or a connection worker stops
    -> mark Remote Control as non-connected promptly
    -> cancel and drain only the current relay connection generation
    -> reconnect the relay WebSocket
    -> keep app-server PID, Unix socket, enrollment, installation_id, and local clients intact

This issue is a solution proposal for the relay transport/liveness layer. It is not intended to cover every possible symptom in #26786. If ordinary local app-server RPCs also stop responding when Remote Control is stuck, that likely points to a broader app-server dispatcher/local transport problem and should be diagnosed separately.

Proposed scope

Handle these cases in codex-app-server-transport Remote Control code:

• relay WebSocket closes or errors;
• relay connection remains open but stops producing pong frames;
• one connection worker exits while another worker is still draining;
• status should not remain connected after the current connection generation is known dead.

Do not handle these in the first transport-scoped change:

• restarting app-server or daemon;
• repairing daemon ownership or "unmanaged app-server" states;
• deleting or replacing enrollment;
• rotating installation_id;
• adding a public reconnect RPC;
• changing app-server protocol/API;
• backend presence expiry while the relay WebSocket is otherwise healthy.

Proposed implementation

Keep the existing long-lived RemoteControlWebsocket state intact:

• enrollment/current enrollment;
• pairing persistence key;
• client tracker;
• outbound buffer;
• subscribe cursor;
• sequence/cursor state;
• desired-state and status publishers.

Treat only the following as per-connection-generation state:

• WebSocketStream;
• reader worker;
• writer worker;
• child cancellation token;
• ping/pong deadlines;
• connection drain timeout.

The minimal behavior change is:

1. When either connection worker stops, immediately publish RemoteControlConnectionStatus::Connecting.
2. Cancel only the current connection generation.
3. Drain or abort the sibling worker using the existing connection shutdown timeout.
4. Return to the existing reconnect loop to establish the next WebSocket generation.

This prevents stale local status from remaining connected after the relay generation has already failed, and it avoids restarting app-server or rebuilding long-lived Remote Control state.

Local regression tests

I have a local patch with deterministic fake-relay tests against current upstream/main:

2c5bc5e284402d2e618c15e396d2e81bcc1ec257

The main regression test exercises:

1. Start Remote Control against a local fake relay.
2. Complete enrollment and establish the first relay WebSocket.
3. Observe connected.
4. Keep the TCP/WebSocket connection open but stop producing pong frames.
5. Advance past the pong timeout.
6. Assert status transitions to connecting.
7. Assert a second relay WebSocket handshake is established.
8. Assert a remote initialize message over the second generation still reaches the transport.

A smaller unit test covers the lifecycle edge case where one connection worker stops while the sibling worker is still stuck during drain. It asserts status becomes connecting before drain completes.

Validation:

just test -p codex-app-server-transport \
  remote_control_transport_reconnects_after_disconnect \
  remote_control_transport_reconnects_after_pong_timeout \
  run_connection_marks_status_connecting_when_worker_stops_before_drain

3 tests run: 3 passed

Acceptance criteria

• A stalled relay connection establishes a new WebSocket generation in the same app-server process.
• app-server PID is unchanged.
• local Unix socket transport is not deleted or recreated.
• existing local TUI/app-server clients are not disconnected by relay recovery.
• status does not remain connected after a relay generation has failed.
• user-initiated Remote Control disable still stops retrying.
• app-server shutdown still interrupts reconnect/backoff promptly.
• enrollment is not recreated.
• installation_id is not changed.
• no public API surface is added.
• existing normal WebSocket close reconnect behavior continues to pass.

Open questions

• Is connecting the preferred transient status when a previously connected relay generation stops, or should this publish errored first and then connecting?
• Would maintainers prefer the first PR to only fix status invalidation and pong-timeout recovery, leaving structured reader/writer exit reasons and generation-id logging to a follow-up?
• For remote-control relay connection silently disconnects after 20–55 minutes without auto-reconnect (WSL2) #26786 specifically, should reporters collect paired results for remoteControl/status/read and a non-Remote-Control RPC such as config/read at failure time, to distinguish relay-layer failure from app-server dispatcher failure?