Skip to content

service (v2): restart spawns serve --service in a silent retry loop, forcing manual process kill #37239

Open
Open
service (v2): restart spawns serve --service in a silent retry loop, forcing manual process kill#37239
Assignees
rekram1-node
Labels
2.0

Description

@dmmop
dmmop
opened on Jul 16, 2026

Summary

opencode2 service restart (and version updates) intermittently triggers a restart loop in the background service: the old process is torn down, then serve --service is spawned ~16 times in a row over ~2.5 minutes, with every attempt dying silently (no error logged) until one finally boots. During the loop all clients are disconnected and the user is forced to kill processes manually.

Environment

  • opencode version: 0.0.0-next-15636
  • OS: Ubuntu 26.04 LTS (Resolute Raccoon), kernel Linux 7.0.0-27-generic (x64)
  • Terminal: ghostty (TERM=xterm-ghostty, COLORTERM=truecolor)
  • Shell: /usr/bin/zsh
  • Install/channel: next channel, installed via npm (@opencode-ai/cli, ~/.npm-global/bin/opencode2). Runtime: Node v22.22.1, Bun 1.3.14
  • Active plugins: orchestator.ts, pushover-notifier.ts, rtk.ts (local ~/.config/opencode/plugins/), opencode-key-rotator (local project plugin), @slkiser/opencode-quota@latest, opencode-vibeguard@latest

Reproduction

  1. Run a background service with several plugins configured (mix of local + remote @latest).
  2. Run opencode2 service restart (or let an auto-update restart the service).
  3. Observe that clients disconnect with error=Transport.
  4. Observe the service manager spawn serve --service repeatedly without any succeeding.

Note: this is intermittent. On another restart the same day (crossing an update 15355→15356) the service came up on the first attempt. This intermittency is consistent with a race condition in the old→new process handoff.

Expected Behavior

service restart stops the old service, waits until it fully releases its resources (port/socket), then starts exactly one new serve --service process that boots cleanly. If a start attempt fails, the manager should log the failure and apply a bounded retry with backoff — not spawn processes indefinitely.

Actual Behavior

The service manager spawns 16 serve --service processes over ~2m 32s, all of which die immediately leaving only a cli starting log line and no boot signals, error, bind/EADDRINUSE, panic, or warning. The 17th process finally boots. The silent death is why the user has to kill processes manually.

Objective evidence — per-run log line counts and boot signals (location services booted / watcher started) during the 2026-07-16 06:50–06:53 incident:

cd9f2122: total_lines=1 booted_signals=0
0d44db5b: total_lines=1 booted_signals=0
788b8389: total_lines=1 booted_signals=0
50430622: total_lines=1 booted_signals=0
78dca9c4: total_lines=1 booted_signals=0
12246f65: total_lines=1 booted_signals=0
c81976bd: total_lines=1 booted_signals=0
572ab573: total_lines=1 booted_signals=0
192f0b68: total_lines=1 booted_signals=0
bc985b64: total_lines=1 booted_signals=0
25d44b97: total_lines=1 booted_signals=0
6c09133d: total_lines=1 booted_signals=0
8b5f572b: total_lines=1 booted_signals=0
66245221: total_lines=1 booted_signals=0
a1234a07: total_lines=1 booted_signals=0
bfe9213e: total_lines=1 booted_signals=0
b3a1a2cc: total_lines=82 booted_signals=17   <-- only this one booted

Additional Context

Log timeline (~/.local/share/opencode/log/opencode.log, trimmed):

06:50:25 event stream disconnected  attempt=1 error=Transport   (x2 clients)
06:50:25 cli starting  args=["service","restart"]
06:50:31 cli starting  args=["serve","--service"]
06:50:31 cli starting  args=["serve","--service"]
06:50:36 cli starting  args=["serve","--service"]
06:50:41 cli starting  args=["serve","--service"]
06:50:41 cli starting  args=["serve","--service"]
06:50:46 cli starting  args=["serve","--service"]
06:51:01 cli starting  args=["serve","--service"]
06:51:01 cli starting  args=["serve","--service"]
06:51:06 cli starting  args=["serve","--service"]
06:51:31 cli starting  args=["serve","--service"]
06:51:32 cli starting  args=["serve","--service"]
06:51:37 cli starting  args=["serve","--service"]
06:51:53 cli starting  args=["service","status"]
06:51:57 cli starting  args=["service","start"]
06:52:02 cli starting  args=["serve","--service"]
06:52:02 cli starting  args=["serve","--service"]
06:52:02 cli starting  args=["serve","--service"]
06:52:12 cli starting  args=["serve","--service"]
06:52:35 cli starting  args=["service","start"]
06:52:40 cli starting  args=["serve","--service"]   <-- run b3a1a2cc, boots OK
06:52:57 event stream connected

Between the 16 failed serve --service starts there is no intervening log output of any kind (verified by filtering out cli starting/update check/event stream lines — result is empty), i.e. the failed workers produce zero diagnostics.

Concurrent (possibly contributing) issue — a config-watch error is logged on every service boot:

level=ERROR message="failed to subscribe" directory=/home/david/.config/opencode
cause="Error: inotify_add_watch on '/home/david/.config/opencode' failed: Not a directory
    at Watcher.acquire (../core/src/filesystem/watcher.ts:134:21)
    at Config.reconcileWatches (../core/src/config.ts:361:12)
    at ServerProcess.start (src/server-process.ts:64:29)
    at cli.server-process.run (src/commands/handlers/serve.ts:10:33)"

~/.config/opencode is a symlink to a real directory (../.dotfiles/ai/.config/opencode), yet inotify_add_watch reports Not a directory — the watcher does not appear to resolve the symlink to a directory during reconcileWatches.

Likely factors that widen the race window: 6–7 plugins are (re)loaded on every start attempt, including remote @latest packages and a local project plugin, lengthening startup.

Requests:

  1. Serialize service restart so the new process only starts after the old one has fully exited and released its listening socket.
  2. Log why a serve --service worker exits (currently silent) — capture exit code / bind errors.
  3. Bound the restart retries with backoff and surface a failure to the user instead of looping.
  4. Fix the symlink-directory resolution in the config watcher (inotify_add_watch ... Not a directory on a symlinked config dir).

Activity

  1. github-actions
    assigned
    rekram1-node
    on Jul 16, 2026

  2. github-actions commented on Jul 16, 2026

    @github-actions
    github-actionsBot
    on Jul 16, 2026 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

    If your issue is specifically about the silent failure with no logs from the 16 failed workers (rather than the general herd/race), it may warrant its own tracking. But it would be worth checking whether #36285 already covers your use case or adding your log evidence there.

  3. opencode-agent
    added
    2.0
    on Jul 16, 2026

  4. opencode-agent commented on Jul 16, 2026

    @opencode-agent
    opencode-agentBot
    on Jul 16, 2026 – with OpenCode Agent
    Contributor

    This was caused by an incompatibility between two versions. Running pkill -f opencode2 should clear the stale process and get things working again. Please let us know if it happens again.

  5. dmmop commented on Jul 16, 2026

    @dmmop
    dmmop
    on Jul 16, 2026
    Author

    Still happening in update to v0.0.0-next-15653
    And some Shell commands breaks background service, opencode2 service status report "stopped", and process keep stale in background

  6. dmmop commented on Jul 16, 2026

    @dmmop
    dmmop
    on Jul 16, 2026
    Author

    Update: the sessions were not lost and the SQLite database is healthy.

    After the forced shutdown / stale-service recovery, the session picker stopped showing the historical sessions for this project. However, the sessions still exist and can be resumed directly with:

    opencode2 -s <session-id>

    The problem is specifically the session-list UI/client path:

    opencode2 api get /api/session

    returns the missing sessions, including their project and directory metadata. The SQLite database also passes PRAGMA integrity_check, and the sessions are present in the main database even when inspected without the WAL.

    /api/session/active is not the relevant endpoint here: it correctly returns only the currently running session. /api/session is the persistent history endpoint, and it returns the sessions that are absent from the UI.

    Environment:

    • OpenCode: v0.0.0-next-15659
    • OS: Ubuntu Linux 7.0.0-27-generic, x86_64
    • Terminal: Ghostty (xterm-256color)
    • Shell: zsh
    • Reproducibility: persistent after the forced shutdown and stale-process recovery

    Expected: the session picker should render the sessions returned by GET /api/session, or at least the project root sessions.

    Actual: sessions are absent from the visual list but remain accessible by ID, and the API returns them.

    This appears to be a client-side session-list state, filtering, cursor, or pagination regression rather than SQLite/WAL data loss.

  7. B67687 commented on Aug 3, 2026

    @B67687
    B67687
    on Aug 3, 2026
    Contributor

    Same silent-retry failure mode observed (2026-08-02, v1.18.11)

    Not the serve --service path, but the same disease: a session loop hit a transient provider error — AI_APICallError: Internal Server Error on deepseek-v4-flash via opencode-go at 10:10:07 — and retried silently forever: ~94% CPU for 2 hours, zero new log lines (last line 10:10:28.567, then nothing until killed at 12:35).

    Collateral: the loop kept writing session state into the shared SQLite WAL store → 5.48GB WAL on a 5.45GB DB (never checkpointed, writer never paused) → every other opencode instance in that project then crashed on boot (native abort at 'booting location services').

    The pattern worth fixing

    1. Silent retry: no backoff, no max-attempts, no WARN/ERROR log, no stop-and-surface. Two hours of invisible CPU.
    2. Unbounded iterations: the loop had 1,499 iterations across 8 sessions in ~3.5h before erroring, then retried forever.
    3. Shared-store collateral: one looping session destabilizes every concurrent instance via the shared DB.

    Our guardrails now: retry_on_errors includes 500 (bounded retries, then surface error), per-session iteration caps, and a WAL size watchdog (alert >200MB). Would love to see equivalent defaults upstream: capped retries with logging on provider errors, and a WAL/size guardrail in the store layer.

  8. rorshopping commented on Sep 5, 2026

    @rorshopping
    rorshopping
    on Sep 5, 2026

    Related Windows datapoint for this thread (v0.0.0-beta-19151, Windows 10.0.26200): after an unplanned shared-service shutdown, I observed three concurrent election contenders in the same second — not a restart loop, a post-crash herd:

    timestamp=2026-09-05T14:13:25.790Z run=1cbf88bb message="cli starting" args="[\"serve\",\"--service\"]" role=cli
timestamp=2026-09-05T14:13:25.791Z run=51294b05 message="cli starting" args="[\"serve\",\"--service\"]" role=cli
timestamp=2026-09-05T14:13:25.790Z run=ce2d19b8 message="cli starting" args="[\"serve\",\"--service\"]" role=cli

    Both TUI event streams dropped with Transport errors ~60ms earlier and reconnected once the third contender won (event stream disconnected ... error=Transport → connected, attempts 0→1). The prior incumbent (run=35b1ef5d, ~1h uptime) had just logged InterruptError: All fibers interrupted on /api/event, and running executions were cancelled with subsequent GET /api/session/<id> → 404s.

    This matches the confirmed phenomenon in #36285 (concurrent callers launch multiple contenders) but as a crash-recovery herd rather than a restart loop, so I'm adding the evidence here instead of opening a duplicate. Happy to share fuller redacted logs if useful.

    Suggested direction if this is ever revisited: single-flight service election — contenders probe for a healthy service (with a little jitter) before and during boot, and a contender that observes a healthy service mid-boot attaches to the winner and exits instead of each one booting a full server. The losers here exited quietly, so the minimal fix is making that loser-path fast and deterministic rather than the broader lifecycle rework #36285 set aside.

  9. AntoineGS
    added a commit that references this issue on Sep 21, 2026

    fix(opencode): point OPENCODE_CONFIG_DIR at the resolved config dir

    c5ffa40
  10. AntoineGS
    added a commit that references this issue on Oct 10, 2026

    fix(opencode): point OPENCODE_CONFIG_DIR at the resolved config dir

    9084a85

  11. Manus-Vindicte commented on Oct 22, 2026

    @Manus-Vindicte
    Manus-Vindicte
    on Oct 22, 2026

    Independent reproduction on Windows (opencode TUI + serve --service), with the full spawn chain caught red-handed via a detached process watcher (20 s polling, no gaps):

    wt.exe "37-Adapter"                    <- terminal window
└─ cmd.exe /c shims\opencode.cmd
   └─ pwsh -File _run-tool.ps1 opencode
      └─ cmd.exe /c opencode.cmd
         └─ opencode.exe               <- TUI, stable for hours
            ├─ opencode.exe serve --service   <- lives ~2 min, dies silently
            ├─ opencode.exe serve --service   <- next one, same fate
            └─ ...

    Observations matching this issue exactly:

    • Each serve --service worker lives ~2 minutes, then disappears without any error output anywhere I can see. The next one spawns immediately. No EADDRINUSE, no panic, no log line marking the death.
    • Client side shows exactly your symptom: Transport: The socket connection was closed unexpectedly → reconnect in ~100 ms → repeat every few minutes.
    • Control experiment: running opencode serve manually in a console stays up indefinitely and serves fine — so the binary itself is healthy; the churn comes from the managed lifecycle around it, not from a crash in serve.
    • Memory angle from serve (1.18.25): MCP child processes accumulate on web-client reconnects until the server OOMs #46035 is visible here too: orphaned MCP child processes (node index.mjs of a local MCP server) survive their dead serve parent as orphans and accumulate.

    Environment: Windows 11, opencode via opencode.cmd shim, TUI launched from a Windows Terminal tab. No Docker, no systemd involved — so the loop is not specific to the Linux service manager path.

    What would help reporters confirm: log the exit code / signal of dying serve workers and serialize restarts (wait for port release before spawning the replacement), as already requested above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

2.0

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions