cli: service restart can silently reuse an unresponsive incumbent #37795

Summary

opencode2 service restart can silently fail to stop a temporarily unresponsive managed service, spawn replacement contenders, then reuse the original process when it recovers. The command therefore has a success path where an explicit restart does not replace the service at all.

This occurred while the incumbent service was blocked by cold Location work. The service registration remained present, but its health endpoint did not answer within the two-second probe window. The restart command treated that as no existing service rather than a stop failure.

Environment

• opencode version: 0.0.0-next-15834
• OS: Darwin 25.5.0, arm64
• Terminal: WarpTerminal, TERM=xterm-256color, COLORTERM=truecolor
• Shell: /bin/zsh
• Install/channel: global Bun install, next
• Managed service: one version-matched service, healthy before the cold request and healthy again afterward
• Active plugins: three local user plugins; names and paths omitted

Reproduction

1. Start a managed service and record its service instance identity and PID.
2. Make the service temporarily unable to answer /api/health for more than two seconds without exiting, for example with a synchronous cold-Location operation.
3. Run opencode2 service restart during that window.
4. Let the original service recover before a contender replaces it.
5. Check the service instance identity and PID after the command completes.

The production incident was observed once. The source branch makes the outcome deterministic for a registered service whose health probe times out during stop and recovers during ensure.

Expected Behavior

• Successful service restart means the registered service was stopped and a different service instance became ready.
• If the command cannot safely authenticate or stop the incumbent, it should fail with a bounded, actionable error.
• It should never silently reinterpret an explicit restart as ensure-running.
• The implementation must continue protecting against stale registrations that point at reused PIDs.
• Logs should state whether stop was accepted, unsupported, rejected, timed out, escalated, or skipped, and whether ensure reused or started a service.

Actual Behavior

Sanitized incident timeline:

18:01:52.738  incumbent finishes a 53.4-second Location acquisition
18:01:54.079  client event stream attempts to connect
18:01:56.081  connection fails with Transport
18:02:05.397  first serve --service contender starts
18:02:11.808  second contender starts
18:02:12.647  explicit service restart command starts
18:02:24.119  another contender starts
18:02:30.081  another contender starts
18:02:40.841  incumbent finishes delayed post-boot work and becomes responsive
18:03:18       health and service status report the original PID

There is no elected replacement server, shutdown record, or PID change. The original server run continues logging afterward.

The restart handler simply sequences stop and ensure at restart.ts:8-15.

Service.stop calls find and does nothing when find returns undefined:

const existing = yield* find(options)
if (existing !== undefined) yield* kill(existing, options)

find is not a registration lookup. It returns only a service that completes an authenticated health probe. The probe uses a two-second timeout and maps every fetch failure to undefined at service.ts:187-227. Consequently, a registered but temporarily unresponsive incumbent is indistinguishable from no service at the stop boundary.

After the no-op stop, ensure is allowed to reuse any compatible service that becomes ready. If the incumbent recovers while contenders are still losing election, the explicit restart converges back to the same process.

Additional Context

Safety constraint

The existing code correctly avoids blindly signaling a PID from an unauthenticated stale registration. Any fix must preserve that defense. A safe design may need stronger process ownership evidence in the registration, an authenticated out-of-band control path, or an explicit error when ownership cannot be verified. The bug is the silent success, not merely the absence of unconditional SIGKILL.

Distinction from related reports

• 2.0: managed-service restart causes reconnect herd and resource spikes #36285 covers reconnect herds and resource spikes when a service is actually replaced.
• cli: managed reconnect waits until fourth failure to ensure service #36581 covers waiting until a later reconnect attempt before calling ensure-running.
• service: self-update orphans background server and hangs every client #37521 covered a stale lock-holding service with a dead listener and was fixed by removing the lifetime flock.

This report is narrower: the registration and process are valid, the incumbent is temporarily unresponsive, and an explicit restart can return to that same incumbent.

Acceptance criteria

• A regression test creates a registered service whose authenticated health endpoint times out while its process remains alive.
• service restart either safely stops that exact instance or exits nonzero with an actionable stop error.
• service restart never runs ensure after an unconfirmed stop.
• A successful restart proves that the previous instance exited and that the new registration has a different instance identity and PID.
• A stale registration pointing at a reused unrelated PID is never signaled.
• Stop and restart expose distinct outcomes for missing registration, corrupt registration, unreachable health endpoint, rejected stop, unsupported stop, graceful timeout, escalation, and successful replacement.
• Contender launches and election outcomes are logged without credentials.
• Passive reconnect and ordinary ensure retain their idempotent reuse semantics; only explicit restart requires replacement.

All URLs, credentials, local paths, instance IDs, and exact PIDs have been omitted.