Skip to content

runLoop fails to exit when client-generated messageID has clock skew ahead of server, causing infinite continuation with <system-reminder> wrap #28618

Closed as not planned
Closed as not planned
runLoop fails to exit when client-generated messageID has clock skew ahead of server, causing infinite continuation with <system-reminder> wrap#28618
Assignees
StarpTech

Description

@clansty
clansty
opened on May 21, 2026

Description

Symptom: When the webui/desktop client and the opencode server run on different machines and the client's clock is ahead of the server's, every prompt triggers a spurious second LLM round-trip after the first response finishes cleanly. In the UI a fresh "thinking…" indicator appears below the completed assistant message and a new (cancellable) request is sent.

Root cause: The runLoop exit condition in packages/opencode/src/session/prompt.ts:1268-1276 uses lexicographic comparison of message IDs (lastUser.id < lastAssistant.id) to decide whether the user message predates the assistant message. Message IDs encode Date.now() in their first 12 hex chars (packages/opencode/src/id/id.ts:51-69). The webui passes a client-generated messageID to session.promptAsync (packages/app/src/components/prompt-input/submit.ts:155-162), so the user message's ID is timestamped with the browser's clock while the assistant message is timestamped with the server's clock. If the browser is even a few hundred ms ahead of the server, lastUser.id sorts after lastAssistant.id, the exit condition fails, the loop enters iteration 2, step++ makes step > 1, and prompt.ts:1400-1416 wraps the user message in <system-reminder>The user sent the following message: …</system-reminder> and fires another API request as if a fresh user message had just arrived.

Evidence from a reproduction (opencode 1.15.5, c5y/claude-opus-4-7-fast, web client on host machine, server in Docker on another host):

Message ID Timestamp encoded in ID DB time_created Note
user msg_e4980c734001BQGJCPTvfloq9Y 1779349636916 1779349634300 client-generated, +2616ms ahead
assistant (finish=stop) msg_e4980bda30015NAyO5tNmnGO1U 1779349634467 1779349634467 server-generated
assistant (aborted, the spurious 2nd request) msg_e4980eab3001KzEUq61ccvKyuG 1779349646003 1779349646003 server-generated

Lexicographic order: bda3 < c734 ⇒ lastUser.id < lastAssistant.id is false even though chronologically the user message was first.

Matching log excerpt:

07:47:14 session.prompt step=0 loop            ← 1st iteration starts
07:47:14 llm ... claude-opus-4-7-fast stream   ← 1st API call (succeeds, end_turn)
07:47:25 session.prompt step=1 loop            ← 2nd iteration entered (BUG)
07:47:33 llm ... claude-opus-4-7-fast stream   ← 2nd API call fired (with <system-reminder> wrap)
07:47:36 session.prompt cancel                 ← user manually cancelled

The second-call request body has messages = [ user (wrapped in <system-reminder>), assistant ], confirming step > 1 && lastFinished was true.

Suggested fix: stop relying on the lexicographic order of (potentially client-generated) ID timestamps to determine intra-session message ordering. Use the server-recorded time_created field, or track "messages created during this runLoop invocation" explicitly, or generate the user messageID server-side and ignore any client-provided one.

Plugins

oh-my-openagent@4.2.3, opencode-pty, @nick-vi/opencode-type-inject

OpenCode version

1.15.5

Steps to reproduce

  1. Run opencode server in a Docker container on host A: opencode serve --hostname 0.0.0.0 --port 4096
  2. Open the web UI in a browser on host B (different machine).
  3. Introduce a clock skew of ≥ ~200ms with host B ahead of host A. Easy way: on host A run sudo date -s "$(date -u -d '3 seconds ago')" (or temporarily stop NTP and let drift accumulate). Verify with date -u on both machines.
  4. In the web UI, start a brand-new session and send any short prompt (e.g. 你的 system prompt 里有什么 / what is in your system prompt). Pick a model whose response naturally finishes with stop_reason: end_turn and uses no tools.
  5. Observe: after the assistant finishes streaming, a new "thinking…" indicator appears below it and a second API request is fired with the original user message wrapped in <system-reminder>The user sent the following message: … Please address this message and continue with your tasks.</system-reminder>.

Expected: runLoop exits after the first response (finish=stop, no tool calls, single user message in session) — no second request.
Actual: runLoop iterates once more, sends a duplicate request with the system-reminder wrap.

The bug disappears as soon as the clocks are re-aligned (e.g. sudo chronyc -a makestep on host A).

Screenshot and/or share link

No response

Operating System

Debian 13 docker container + msedge 148.0.3967.70 as client

Terminal

No response

Activity

  1. github-actions
    assigned
    StarpTech
    on May 21, 2026

  2. github-actions commented on May 21, 2026

    @github-actions
    github-actionsBot
    on May 21, 2026 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

  3. kheerand commented on May 24, 2026

    @kheerand
    kheerand
    on May 24, 2026

    Confirmed hitting the same bug from the local single-machine angle — no client/server split, no clock skew, just millisecond-resolution ID collisions on the same host.

    I filed #28986 before finding this issue. Adding my quantification here since it may help validate the fix:

    Local collision evidence

    The same sA.id < L.id failure happens even when everything runs on one machine. The ID generator (ascending()) produces non-monotonic IDs when messages are created within milliseconds of each other.

    Example (same-host, same process)

    Message ID Created (ms) Role
    User question msg_e55166ef1001... 1779543994009 user
    Assistant response msg_e55166ed1001... 1779543994065 assistant

    56ms apart, but the user message ID prefix e55166ef sorts after the assistant ID prefix e55166ed. Exit condition fails, loop continues.

    Database quantification (opencode 1.15.6, local SQLite)

    Measured from the full session database:

    • 16,069 user→assistant message pairs created within 1 second of each other
    • 453 pairs (2.8%) have inverted IDs
    • 143 orphan continuations (self-replies) detected across 36 sessions
    • The model stops with finish: "stop" and zero tool calls, but the loop continues

    What it looks like to the user

    AI: No. It's just a cron job... Want me to disable it?

AI: Yeah, exactly right. The heartbeat is just a cron job with extra steps... Let me disable it.

AI: Heartbeat disabled. Script''s still there if we ever need it.

    Three consecutive AI responses to a single user question — the AI treats its own previous text as new context and continues.

    The suggested fix is the same as noted here: use time_created timestamps instead of ID ordering in the exit condition. Happy to test a fix build if one lands.

  4. Spanjer1 commented on May 26, 2026

    @Spanjer1
    Spanjer1
    on May 26, 2026

    I encounter this as well with @openchamber/web -> opencode serve on a separate host. Server clock was a couple minutes behind the browser, agent keeps responding without ending. The fix for me was just making sure the clock is in sync.

  5. ghost marked <system-reminder> wrapping fires on every tool-call iteration in follow-up messages, derailing model behavior #29659 as a duplicate of this issue on May 28, 2026

  6. ualtinok commented on Jul 12, 2026

    @ualtinok
    ualtinok
    on Jul 12, 2026
    Contributor

    Confirmed on v1.17.18 through session.promptAsync against the live server. This reproduction requires no clock skew: an external plugin supplied content-hash message IDs that satisfy the public schema but do not preserve OpenCode's ordering.

    Evidence

    The first admitted user message used this caller-supplied ID:

    msg_f92303e81b8914b7

    It is 20 characters long. OpenCode-generated messages in the same session used the normal 30-character monotonic form, for example:

    msg_f55b3c1c5001...
msg_f55baa17d001...
msg_f55bb2f59001...

    The custom msg_f923... sorts after every generated msg_f55... message. The assistant repeatedly finished with finish="stop", but lastUser.id < lastAssistant.id remained false.

    The plugin's durable execution ledger and server logs agreed exactly:

    1. The first promptAsync call entered one run loop with contiguous steps 0..56, then the session was aborted.
    2. A second call 11 minutes later used msg_614a2f7eceac0ea0 and entered one run loop with contiguous steps 0..17, then it was aborted.

    The second ID sorts below the first custom ID. Because MessageV2.latest() selects the user with the lexicographically greatest ID, the first injected user remained "latest" even after the second injection and ordinary user messages arrived. Every generated assistant stayed parented to the first injected message.

    This rules out request retries or runner splitting: exactly two admitted calls produced exactly two runaway loops.

    Current source contract

    On current dev:

    • MessageID validation only requires the msg prefix.
    • MessageV2.latest() selects messages using lexicographic ID order.
    • SessionPrompt.runLoop() uses lastUser.id < lastAssistant.id as its terminal exit condition.

    The API therefore accepts IDs that violate an undocumented chronology invariant relied upon by the runtime.

    Impact and mitigation

    This produced repeated paid provider calls after terminal answers and continued until explicit abort. The plugin mitigation is to stop forwarding caller-supplied IDs and let OpenCode generate them.

    Host-side, opaque IDs should not determine transcript chronology. Binding completion to parentID or a server-owned sequence would also cover the clock-skew case originally reported here; validating only the ID length/shape would not.

  7. github-actions commented on Sep 11, 2026

    @github-actions
    github-actionsBot
    on Sep 11, 2026 – with GitHub Actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

  8. github-actions
    closed this as not plannedon Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions