[service]: managed service stalls, watchdog recovery loses sessions (404 "session not found"), restart stampede races the service slot (2.0.25 vs 2.0.26) #54181

Activity

1. github-actions

   assigned

   rekram1-node

   on Oct 9, 2026
2. opencode-agent commented on Oct 9, 2026

   

   opencode-agentBot

   on Oct 9, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report and the measurements. I tried to reproduce this with 2.0.26 on Linux, and none of the problems showed up:

   • Session loss: I created sessions through the API and SIGKILLed the server right after the create returned. Both sessions were still in session_v2 in opencode.db, so new sessions are written to disk before the server responds.
   • Event-loop blocking: a real shell tool call that blocked for about 25 seconds on a network connect to an address that never answers (like your ssh -o ConnectTimeout=15) did not slow the server down. Polling GET / every second during the call, every response took a few milliseconds.
   • Hanging MCP servers: a remote MCP server that never replies, and one on a hostname that doesn't resolve, didn't block the server either.
   • Restart race: starting 6 opencode serve --service processes at once left exactly one running, and service.json pointed at it.

   So the trigger is probably something in your setup I couldn't copy: macOS, Desktop 2.0.25 together with the 2.0.26 CLI, your MCP servers (~450 tools), or the VPN DNS. These would help narrow it down:

   1. From your log, the lines just before one of the long silences, e.g. the 194 s gap after 17:09:14. Is it MCP tool listing, a config or plugin reload, or the file watcher?
   2. Does a stall still happen with the timewebCloud MCP server disabled, or with all MCP servers disabled?
   3. Does it still happen when only the 2.0.26 CLI is used, without Desktop running its bundled 2.0.25?
   4. For the lost sessions: are OPENCODE_DB or any other OPENCODE_* variables set for Desktop or for your shell? Does the ses_edf7c931... ID appear anywhere in your log before the recovery (e.g. a POST /api/session response), and in which run? A session that is completely missing from the DB, with no messages, could mean it was created against a different database file.
   5. A sample <pid> 10 -file stall.txt of the serve --service process during a stall would show exactly what is blocking it.

   Related issues, which may cover parts of this: #52049 (the stall watchdog replaces the service and aborts sessions; fix proposed in #52733), #51216 (service SIGKILLed when /api/info is blocked), #37794 (service blocked for 53 s), #37795 and #48588 (unresponsive service and watchdog kill loops).
3. opencode-agent

   added

   severity:high

   on Oct 9, 2026
4. vkobazev commented on Oct 10, 2026

   

   vkobazev

   on Oct 10, 2026

   Author

   More actions

   Thanks for the quick and thorough triage. Answers below, numbered as in your reply. All timestamps are UTC (the machine is UTC+3).

   1. What precedes a long silence (the 194 s gap, 2026-10-09 17:09:14 -> 17:12:28)

   Right before it: a tool call spawn AND a file-watcher resubscription burst. The whole minute 17:09 has 76 watcher lines (3 bursts); the silence itself has ZERO log lines of any kind (no MCP listing, no config/plugin reload, no watcher), then watcher lines resume at 17:12:

   17:09:08.861 spawning process: /bin/zsh -c "ssh -o ConnectTimeout=15 <host> 'bash -s' ..."
   17:09:08.957-17:09:14.141  watcher stopped/subscribe/started x76
             (backend=fs-events, paths: ~/.claude/skills, ~/.agents/skills, <project>/.claude/skills, ...)
   17:09:14.141 watcher started ... <- last line before the gap
      [194 s: no log output at all, HTTP unresponsive]
   17:12:28.371 spawning process: next tool call <- service wakes up
   

   Watcher lines per minute around the gap: 17:08 -> 0, 17:09 -> 76, 17:10 -> 0, 17:11 -> 0, 17:12 -> 38. So watcher churn appears in bursts around tool executions, not during the silence. The tool call here is one of several (ssh -o ConnectTimeout=15, 4 of them over ~6 minutes), each followed by a 20-194 s silence. The service watches /, /Users, the home dir and the project dirs with type=entries watchers plus fs-events on the skills dirs.

   2. Does it still stall with timewebCloud disabled? - Yes, measured today

   timewebCloud has been disabled in the global config since 2026-10-09 17:45 UTC, and our two custom plugins were fixed in between (see the side finding below). Polling GET / every 2 s just now (2026-10-10 13:41-13:42 UTC, one session actively running local tool calls):

   30 probes: 14 timeouts  (healthy answers take ~2 ms; failing ones get nothing for the full 2 s)
   

   Earlier today with the same setup: 4/15 timeouts. So it reproduces without timewebCloud. A full "all MCP disabled" run needs a quiet window (a session is active right now); happy to do that A/B on request.

   3. Version mix - reproduces on 2.0.26 only as well

   The current managed service is 2.0.26 (service.json: version 2.0.26; Desktop self-updated to 2.0.26 overnight, standalone CLI is also 2.0.26). Stalls continue in this all-2.0.26 state with the same signature. Yesterday the same stalls happened under the 2.0.25-service + 2.0.26-clients mix. The mix amplified restarts (the stampede in the report) but the stall itself is version-independent here. A CLI-only (no Desktop) test is possible on request.

   4. Lost sessions: DB and env

   • No OPENCODE_* variables in the shell or in any rc file. No second opencode database exists on disk (checked Application Support/ai.opencode.desktop, ~/.opencode, ~/.local/share/opencode, ~/.config/opencode - the only opencode DB is ~/.local/share/opencode/opencode.db).
   • ses_edf7c931... appears in the log ONLY as 20 x 404 responses between 11:59:17.066 and 12:00:24.432 (runs 9f9c5d25 and 37807470, i.e. the first seconds after the watchdog recovery at 11:59:11). There is no creation record - but note the log contains no session-creation requests at all: the only logged POSTs in the whole 6-day file are /api/rpc/experimental.browser/attach (53), /api/location/reload (1) and one /experimental/session/.../wait. Creates are simply not visible in the log.
   • Against the "different database" theory: the missing IDs sort between surviving sessions created in the same window (ses_edf77b197ffe... at 14:59:26 and ses_edf97c0adffe... at 14:24:25 local), which ARE in session_v2 of the same DB. They look like same-DB sessions that vanished, not cross-DB ones.

   5. sample during a stall - already captured

   I have two captures of the serve --service process during stalls (auto-triggered: sampling starts the moment GET / times out). Main thread call graph, 6 s capture during a freeze (4579 samples):

   4579 Thread_144287326  DispatchQueue_1: com.apple.main-thread (serial)
   + 4579 start (in dyld) + 7184
   +   4579 ??? (in opencode-cli) + 0x468c
   +     4579 ??? (in opencode-cli) + 0x54f0
   +       4579 ??? (in opencode-cli) + 0x14dbe38
   +         4315 ??? (in opencode-cli) + 0x15c4a8
   +         ! 4217 ??? + 0x5e28b4
   +         ! : 4148 ??? + 0x33ca20 -> 0x33e410 -> kevent64  (in libsystem_kernel.dylib)
   

   The main thread is idle in kevent64 (91%) during the stall - exactly as in a healthy 12 s sample (96% kevent64, same frames). A 45 s capture covering a 16 s stall shows the same distribution (99.4% in the same frame as healthy). No blocking syscall, no getaddrinfo, no sqlite frame: the event loop simply doesn't serve HTTP while the window is frozen. All 28 threads parked as usual in psynch_cvwait/kevent/read. That's why I suspect the HTTP/accept side (or request-queue serialization) rather than a blocked worker. Happy to re-sample with other flags if useful.

   Side finding on our side (fixed, but not the root cause)

   Our custom tool-search plugin did a synchronous read/parse/stringify/write of a 3.2 MB state file from the context hook (every model request) and the execute.after hook (every tool call) - about 25-30 ms of event-loop blocking per call. It also never removed stale catalog entries (226 tools of a disabled MCP server stayed "searchable"), which made the model call non-existent tools. We patched it (debounced + atomic writes, catalog pruning) and the stalls persist, so it was an amplifier rather than the cause. Possibly worth documenting that plugin hooks run synchronously on the hot path.

   Related issues

   #52049 (watchdog replaces the service and aborts sessions) matches our session loss exactly - the log line before the loss is literally Background service is unresponsive; recovery cannot preserve persistent terminals. Data loss is the most painful part for us, so #52733 landing would already be a big win.

   What we can run next (pick any)

   1. A/B with ALL MCP servers disabled for 5 minutes (needs a quiet window).
   2. Same stall test WITHOUT Desktop (CLI only).
   3. More sample captures during a stall with additional flags, or a sysdiagnose.
   4. A minimal repro on this machine: a separate serve instance + headless opencode run with a blocking shell tool call while polling GET / - to see if it reproduces without our config/plugins at all.
5. opencode-agent commented on Oct 10, 2026

   

   opencode-agentBot

   on Oct 10, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed follow-up. The sample captures help a lot.

   I tried again with 2.0.26 on Linux, this time copying the pattern you described before each silence. The project had skills in .claude/skills, .agents/skills and .opencode/skills. A session made four real shell tool calls over about a minute: slow ones (a 15 s connect to an address that never answers, like your ssh -o ConnectTimeout=15), plus calls that created and deleted 20 skill folders. You can see the skill watcher reloading in the TUI. Polling GET / every second for 150 s, all 148 requests answered within a few milliseconds, with no timeouts.

   Your samples show the main thread idle in kevent64 while HTTP stops answering, so nothing is keeping the CPU busy. On macOS the service also uses the fs-events watcher backend, including watchers on /, /Users and your home folder. On Linux it uses a different backend (inotify), so the most likely trigger can't be recreated here. This needs a maintainer with a Mac. Of the runs you offered, these would help most:

   • Your option 4: a separate opencode serve with no custom config or plugins and a blocking shell tool call, while polling GET /. It would show whether the stall happens without your setup.
   • Your options 1 and 2: whether the stall still happens with all MCP servers disabled and Desktop closed (CLI only).

   The lost sessions come from the watchdog killing an unresponsive service, which is the problem tracked in #52049 (proposed fix in #52733). I'm keeping that part in the summary for maintainers.
6. vkobazev commented on Oct 10, 2026

   

   vkobazev

   on Oct 10, 2026

   Author

   More actions

   Follow-up with a full A/B matrix on this machine (macOS 26.3.1, arm64, 2.0.26 standalone binary). Same protocol everywhere: background poll of GET / every 0.5 s (1 s timeout) while one agent runs a blocking shell tool call (sleep 45) via opencode run --server <url> --auto.

   Result: the stall needs the FULL real environment; every isolated piece is clean

   #	Config dir	Data dir	Mode	Tool call	Probe failures
   1	minimal (1 provider, no plugins, no MCP)	fresh	serve --port	sleep 45	0 / 233
   2	minimal	fresh	serve --port	2x sleep 45, 3 concurrent agents, 4 locations (incl. $HOME)	0 / 186
   3	+ our 4 plugins (tool-search, mem0, file-manager, claude-parity)	fresh	serve --port	2x sleep 45	0 / 190
   4	minimal + all 8 MCP servers (6 connected)	fresh	serve --port	sleep 45	0 / 137
   5	full config copy (providers+mcp+plugins+agents+compaction)	fresh	serve --port	sleep 45	0 / 145
   6	full config copy	copy of the real 116 MB opencode.db + state files	serve --port	sleep 45	0 / 130
   7	the real ~/.config/opencode (AGENTS.md, node_modules, memory, tools.config.json...)	fresh	serve --port	sleep 45	0 / 116
   8	full config copy	copy of the real DB	serve --service	sleep 45	0 / 121
   -	real ~/.config/opencode	real ~/.local/share/opencode	serve --service	sleep 45	81 / 220 (37%) and, repeated after a service restart, 26 / 72 (36%)

   So it is NOT the plugins, NOT the MCP servers, NOT the config schema, NOT the DB size/contents, NOT concurrency/locations, and NOT the --service mode per se. It reproduces only when the process runs with the real config dir + real data dir + real state dir together. Also note test 6 proves a 116 MB DB with 258 sessions is fine on a fresh process.

   Additional facts from the live instance

   1. It stalls while completely idle. My stall trap (probe every 0.5 s, auto-sample on first timeout) caught freezes with no sessions, no clients, no tool calls at all (e.g. 17:15:52 local). The correlation with tool calls earlier was because that is when we were looking.
   2. During a stall the process is asleep, not busy: ps shows CPU 0.2-0.5% and state S at the exact moment GET / times out. A sample capture taken during an idle stall (8 s, 6103 main-thread samples) shows the main thread not in any blocking syscall - no getaddrinfo, no sqlite, no file I/O - and only 3 kevent64 leaf samples for the whole window (compare a healthy capture: 96% of main-thread samples are in kevent64 waiting). So the loop is neither computing nor parked in the poll; it looks like it processes a trickle of internal work and only rarely reaches accept.
   3. The failure pattern under constant polling is almost rhythmic: once a stall starts, roughly every other probe fails (0.5 s cadence -> ~1 accept per second vs. hundreds per second when healthy). Failed probes leave half-open sockets behind, so the backlog drains slowly.
   4. During long silences in the log (e.g. the 194 s gap) the log goes completely quiet - no watcher lines, no MCP, no config reloads - while watcher churn appears in bursts around activity (76 watcher lines in the minute before the gap, 0 during it). Watchers in the real environment include type=entries on /, /Users, $HOME and each project dir.
   5. Client-side amplifier we already fixed on our side: 4 stuck CLI processes held 82-121 sockets each (422 in SYN_SENT) because reconnects have no backoff or connection cap. This makes each stall look longer than it is.

   Remaining deltas between test 8 and the real instance

   Everything above isolates the trigger to the real directories as a whole. Not yet isolated separately: the real state dir (~/.local/state/opencode: prompt-history.jsonl 5.2 MB, locks, pty) and the non-DB contents of the real data dir (shell/, snapshot/, storage/, tool-output/, opencode-perf.log, and the 35 MB opencode.log that is being appended to constantly). Happy to bisect these two next if it helps - or run whatever instrumented build you want on this machine.

   Given #52049/#52733 cover the data-loss half (watchdog kills the service and aborts sessions), this report is mainly about the stall half: it is reproducible here 36-37% of the time on the real environment and 0% in every isolated configuration.
7. vkobazev commented on Oct 10, 2026

   

   vkobazev

   on Oct 10, 2026

   Author

   More actions

   Root cause found - and it is not in the config, plugins, MCP, DB or app logic: the stall is bound to the accumulated kernel-level TCP state of one specific port (127.0.0.1:49374). The opencode process is completely healthy during a stall; the OS simply stops completing TCP handshakes to that destination.

   Captured during a sustained stall (4+ consecutive probe failures)

   lsof:
     opencode 83xxx  127.0.0.1:49374 (LISTEN)                     <- listener alive
     curl     84xxx  127.0.0.1:60504->127.0.0.1:49374 (SYN_SENT) <- client stuck in handshake
   netstat -anv:
     tcp4 0 0 127.0.0.1.49374 *.* LISTEN   Recv-Q=0 Send-Q=0      <- accept queue EMPTY
   ps:
     pid 80649  0.0% CPU  state S                                  <- process asleep
   probe curl during the capture: no answer for 6 s
   

   So the SYN never completes and never lands in the accept queue - the process cannot be blamed: its main thread is 99.2% in kevent64 during the stall (6103-sample capture), i.e. idle and waiting for events, CPU 0%.

   The decisive A/B: same machine, same real environment, different port

   Instance	Port	Probe failures
   real config + real data (116 MB DB) + real state	49374	36-37% (81/220, then 26/72, both reproducible)
   the very same real directories	48888	0 / 143
   after moving the service to a fresh port (--port 49474)	49474	0 / 104 and counting

   During a stall on 49374 a control listener (python http.server) on another port answered in 6 ms - so the OS TCP stack is fine; the failure is specific to that one destination port.

   Why restarts never fixed it for the user

   The port is sticky: even after deleting service.json, the next opencode serve --service bound 49374 again (the port seems to be derived deterministically in the binary - the string is right next to the cli.service-config code). Every restart therefore re-entered the same degraded per-port state. Starting with an explicit --port 49474 fixed it instantly.

   What degrades that port

   At capture time: 8000+ sockets in TIME_WAIT associated with 49374 (21942 TIME_WAIT machine-wide), and earlier in the day a single stuck CLI client left 82-121 half-open sockets each (422 in SYN_SENT at once) because reconnects have no backoff or connection cap. That is a lot of per-destination TCP history for one 127.0.0.1:port tuple.

   Environment note (possible co-factor): macOS 26.3.1 with three overlay/VPN systems active - a Tailscale network extension (io.tailscale.ipn.macsys.network-extension), NetBird and AmneziaVPN. A network extension can hold per-flow state; the control-port test shows the stack otherwise works.

   Recommendations

   1. Do not rebind the previous port on service start - pick a random free port and write it to service.json. This is what made the problem permanent for the user: each restart re-entered the degraded state.
   2. Client reconnects: exponential backoff + a connection cap (the 422-half-open-socket storm made this much worse).
   3. Keep cli(win): 45s event-stream idle watchdog restarts the managed service, aborting all sessions/subagents #52049/fix(client): reconnect transient stalls instead of replacing the service (#52049) #52733 (watchdog must not lose sessions) - that remains the data-loss half.

   Current state on this machine: service runs on 127.0.0.1:49474, 104 consecutive probes without a single timeout (max latency 5 ms), where the old port failed ~36% of probes minutes earlier.
8. vkobazev commented on Oct 10, 2026

   

   vkobazev

   on Oct 10, 2026

   Author

   More actions

   Final piece of the puzzle - and it is not an opencode bug at all: the port itself becomes poisoned at the kernel level by TIME_WAIT 4-tuple collisions on loopback.

   The mechanism

   1. opencode's HTTP sends Connection: Keep-Alive: timeout=5 - so the server closes every idle client connection after 5 s. The closing side lands in TIME_WAIT. Out of 8006 TIME_WAIT sockets associated with 127.0.0.1:49374, 7556 are server-side (src=49374).
   2. The service port is sticky (rebinds 49374 on every restart), so all those tuples pile up on one destination port. In the capture, the opposite ports covered 8006 unique ports = 48.9% of the entire ephemeral range (49152-65535).
   3. When a client (curl/Desktop/CLI) picks a source port that is still in TIME_WAIT for that same 4-tuple, the kernel protects the old socket and does not complete the new handshake -> the client hangs in SYN_SENT with an empty accept queue and a perfectly healthy server process (which matches every capture in my previous comments).
   4. Predicted collision probability from the port occupancy: ~49%; observed probe failure rate: 36-37%. Same magnitude (the exact ratio depends on kernel port-pick/retry behavior).

   Decisive control experiment (today, 18:10 local)

   The opencode service was moved to a fresh port (49474) 14 minutes earlier. Then I bound a plain python http.server to the old port and polled both with identical probes:

   fresh python listener on the old port 49374: 10 failures of 40   (25%)
   opencode service on the fresh port 49474:    0 failures of 40   (0%)
   

   A different program on the old port fails exactly the same way -> the port is poisoned in the kernel, independent of opencode.

   Also interesting: the 8013 TIME_WAIT sockets on 49374 did not expire after 14 minutes without traffic (MSL here is 15 s, yet they persist - macOS keeps them far longer in practice). That is why the user's problem lasted for days and why every restart (same sticky port) kept failing.

   Why Linux did not reproduce

   Linux accepts a SYN to a TIME_WAIT tuple when it advances the sequence (RFC 6191, and tcp_tw_reuse), and the test had low connection churn. On macOS loopback with a fixed destination port and heavy keep-alive churn the collisions are frequent.

   Recommendations (revised with this mechanism)

   1. Pick a random port on each service start (as suggested before - now with a concrete reason: avoids any port ever getting "hot").
   2. Keep client connections alive / reuse them; the 5 s server-side keep-alive close is what manufactures TIME_WAIT sockets on every idle disconnect. Consider letting the server keep idle connections longer (30-60 s) or making clients close instead of the server.
   3. The client reconnect storm (no backoff, 82-121 sockets per stuck client) multiplies the churn - the backoff fix from before now has double value.
   4. Data-loss half is still cli(win): 45s event-stream idle watchdog restarts the managed service, aborting all sessions/subagents #52049/fix(client): reconnect transient stalls instead of replacing the service (#52049) #52733.

   Environment: macOS 26.3.1 (arm64), MSL=15 s, somaxconn=128, three overlay/VPN systems installed (Tailscale network extension active, NetBird, AmneziaVPN) - probably not required for the mechanism, but they add connection churn of their own.
9. LZmx commented on Oct 26, 2026

   

   LZmx

   on Oct 26, 2026

   More actions

   Same stall + stampede on 2.1.4 (macOS), with a distinct self-shutdown path

   Adding a 2.1.4 data point because the recovery loop here ends in a different
   message than the one in the report: the running service does not see its
   registration replaced, it sees the file gone (ENOENT) and terminates
   itself. No OS kill and no crash report were involved.

   Environment

   • opencode 2.1.4, channel latest, standalone ~/.opencode/bin/opencode
   • macOS 26.x / Darwin 27.0.0 arm64, iTerm2, zsh
   • 1 local plugin, no MCP servers configured
   • data dir ~/.local/share/opencode, opencode.db 4.74 GiB (1.24 M pages,
     652 k free pages), 1,088 sessions / 65,406 messages / 280,839 parts
   • several TUI sessions open against the one shared service

   What the user sees

   The TUI stays on Restarting service… / Your session will resume automatically.
   for ~40 minutes. All open sessions show it at once. Nothing reconnects and
   nothing new is persisted. Only a machine reboot cleared it.

   For reference, that overlay is just a connection indicator in the TUI: it is
   shown once connection.status() !== "connected" for 1 s (reconnecting) or
   5 s (any other state). It never initiates a restart itself, so it stays up for
   as long as the client cannot reach a service.

   Timeline (UTC)

   09:40:28  role=cli     event stream disconnected
                          error="Transport: InvalidHTTPResponse fetching /api/event"
             role=server  Sent HTTP response GET /api/event status=200
                          <- server kept executing tools and spawning git
                             snapshot processes until 09:42:42, then silence
   09:47:23  role=cli     cli starting args=["serve","--service"]   <- first retry
   ...       68 x `cli starting` over ~45 minutes
   09:49:44  role=cli     cli process failed
                          "Managed service port 49374 on 127.0.0.1 is already in
                           use by another process"
   09:57:40  role=server  WARN "managed service registration check failed; shutting down"
                          NotFound: FileSystem.readFile(
                            ~/.local/state/opencode/service.json)
                          servicePID=88171
   09:57:49  role=cli     cli process failed  cause=Cause([Interrupt(46)])
   10:02:32  role=server  same ENOENT self-shutdown (servicePID=88474)
   10:07:50  role=server  same ENOENT self-shutdown (servicePID=89190)
   10:20     machine rebooted
   10:35     fresh 2.1.4 install, single service, healthy
   

   No DiagnosticReports entry for opencode/bun, no jetsam kill, and
   PRAGMA integrity_check on the DB returns ok — so the original process was
   not killed by the OS and the DB was not corrupt. During the loop each spawned
   service ran for roughly 80–140 s before reporting EADDRINUSE, so every
   doomed attempt also paid the full startup cost (config, agents, skills, ~430
   file watchers, MCP, DB open).

   Keep-Alive: timeout=5 is still present on responses from 2.1.4 (measured
   after the reboot), so the server-side TIME_WAIT mechanism described at the end
   of this thread is still in play. I could not capture TIME_WAIT counts during
   the incident; immediately after the reboot the port had 0.

   Why recovery cannot converge

   1. The service is gone or unreachable, so every client tries to spawn its own
      serve --service.
   2. The port is still held (stale listener or poisoned 4-tuple space), so
      replacements fail with EADDRINUSE — after a full startup.
   3. A process that does win the port deletes/rewrites
      ~/.local/state/opencode/service.json. Whoever observes the file missing
      treats that as "I am no longer the registered service" and exits
      (Interrupt(46)), which hands the port back to nobody.
   4. Clients never get connected, so the overlay never clears.

   There is no eviction of a stale port holder and no single-writer guarantee for
   the registration file.

   Suggestions

   • Serialize service startup with an exclusive lock (flock/lockfile in the
     state dir). One process starts; the rest wait and attach.
   • Treat a missing/unreadable service.json as "re-register", not as
     "another service took over, shut down". A transient ENOENT during a restart
     should not kill a healthy listener.
   • Take ownership of the port (or probe it) before the expensive startup, so a
     2-minute boot does not end in EADDRINUSE. If the registered PID no longer
     answers /api/info, evict it instead of refusing to bind.
   • Client reconnect: exponential backoff and a connection cap, so N clients
     cannot stampede N services.
   • Reconsider the 5 s keep-alive close (or bind with SO_REUSEADDR/reuseport)
     to keep the fixed port from accumulating server-side TIME_WAIT tuples.

   Related: #54267 (no eviction of a previous listener), #58389 (commands starting
   a second service).
10. iceteaSA commented on Oct 26, 2026

    

    iceteaSA

    on Oct 26, 2026

    More actions

    Same watchdog kill on 2.1.4 (Linux), caused by load on the main thread

    Linux x86-64, standalone 2.1.4, one shared service, about 15 TUI clients, many plugins.

    What occurs

    1. The JS main thread of the service is busy (we measured 97-100% of one core) and does not answer requests for some seconds. The process stays alive and continues to write its log (about 80 lines each second, with no interruption).
    2. The clients log event stream disconnected ... error="Event stream stalled".
    3. Approximately 10 s after that, the ensure probe of a client times out 3 times in sequence (2 s each). That client stops the service with SIGTERM, then SIGKILL. Two or three clients then start serve --service in the same 10 ms.
    4. The new service starts again all the sessions that it stopped, at the same time. This puts load on the main thread again. We saw two restarts 3 minutes apart.

    Code (current dev 0749024, same as 2.1.4)

    • packages/client/src/effect/service.ts:75-87: after 3 probe timeouts in sequence, the client stops the service and starts a new one.
    • packages/client/src/service-timing.ts:18: requestTimeout: 2_000.

    Thus the client kills a process that is slow but alive after approximately 6 s. The probe cannot tell a busy service from a hung service.

    One cause of the load, for others who see this

    On each refresh, the skill config plugin removes all of its skill directory watchers, subscribes again and examines each skill folder again (packages/core/src/config/plugin/skill.ts:149-191). We saw 66-140 watcher subscribe lines each second for ~/.claude/skills and ~/.agents/skills. This is the same pattern as the watcher stopped/subscribe/started x76 lines in an earlier comment here. When we removed those two directories, the lines stopped immediately and the main-thread load decreased.

    Possible repairs

    • Do not kill a service that has a live pid only because 3 probes timed out. Use a much longer limit (minutes), or make sure that the service does no work (no new log lines or events) before the kill.
    • Let one client own the recovery, so that clients do not race to start a replacement.
    • If possible, answer the health probe without a wait for the busy main thread.