Skip to content

shell: plugin suffix rebuild drops background completion notifications #56729

Description

@521ox

shell: plugin suffix rebuild drops background completion notifications

Summary

A background shell can finish successfully without notifying its Session after Plugin.activate reconstructs the suffix containing opencode.tool.shell. The Job survives and retains its completed output, but the completion observer belongs to the old plugin activation scope and is cancelled when that scope closes.

This report concerns that specific suffix-reconstruction path, not every reload. An unchanged-activation control delivers its notification and clears its background marker normally.

Environment

  • OpenCode source package version: 2.0.32; in-process Core fixture, not an installed CLI/server reproduction.
  • Tested custom checkout: 72d0dce8ea437ea061e0680110fa19a81aa118fd.
  • Fixed official V2 comparison: 63c1a27af5a391d9caa55b82694f496911e1859b.
  • Runtime / OS: Bun 1.4.2 (744846f84), Windows 10.0.22631, win32 x64.
  • Terminal: not applicable; no TUI was involved.
  • Shell execution: controlled process handle using the existing Core ChildProcessSpawner fixture pattern; no external shell was launched.
  • Plugins: fixture generations [audit.prefix, opencode.tool.shell], then [opencode.tool.shell]; no user plugins or configuration were loaded.
  • Isolation: private HOME, ProgramData, ALLUSERSPROFILE, APPDATA, XDG, config and temporary directories; in-memory Config and SQLite; all fetch destinations blocked, with zero attempts recorded. No real provider, model request, user database, private content or credentials were used.

Reproduction

Build an offline Core fixture with the real Plugin, ShellTool, Tool, Shell, Job, Session and KV services. Reuse testEffect, tempLocationLayer, tempGlobalLayer, hostEnvironmentLayer and the Tool execution helper from packages/core/test. Use Config.testLayer, SessionExecution.noopLayer, offline models and a controlled spawner handle, as in shell-retention.test.ts.

  1. Create a Session in the temporary Location. Register a no-op prefix plugin followed by the real ShellTool generation through Plugin.activate.

  2. For the ShellTool generation, capture its required services without capturing Scope, as PluginInternal.list does. Let actual plugin activation supply its child Scope.

  3. Execute the real tool snapshot with this input:

    { "command": "echo fixture", "background": true }

    Keep the controlled process blocked on a Deferred. Confirm the tool returned status: running, the Job is running, the Session has no completion notice, and jobs.pendingBackground contains one running marker.

  4. Await plugins.activate([shellGeneration]), removing the preceding no-op generation. This reconstructs the shell plugin suffix while leaving the Shell service and Job alive. Instrumented activation counts become 2 starts / 1 closed scope.

  5. Release the process handle with output fixture-output\n and exit code 0. Await jobs.wait({ id: shellID }), then inspect the Session inbox and jobs.pendingBackground.

  6. Control: use the same sequence but call plugins.activate([prefixGeneration, shellGeneration]) without changing either generation. Wait for the real Job.completeBackground acknowledgment before inspecting the final control state.

The rebuilt case closes the old activation before releasing the process. It does not depend on a timing race or an arbitrary sleep to infer that the observer is gone.

Expected Behavior

An already-backgrounded shell that finishes successfully should admit one completion notification and acknowledge its background marker, even when its tool plugin was reconstructed while the independently owned Job remained alive.

Actual Behavior

Observation Unchanged activation Reconstructed suffix
ShellTool starts / closed scopes 1 / 0 2 / 1
Final Job status completed completed
Final Job output fixture-output\n fixture-output\n
Session shell notifications 1 0
pendingBackground empty 1 completed marker containing the output

The unchanged control passes. The reconstructed case fails the notification assertion: Expected length: 1; Received length: 0. The retained marker is evidence that the output is not lost; the broken behavior is automatic notification/acknowledgment.

Additional Context

Relevant relative source locations:

  • packages/core/src/tool/plugin/shell.ts:108,157-189,254-256: captures the plugin Scope and forks notifyWhenDone into it.
  • packages/core/src/plugin.ts:108-138: closes and reconstructs the changed activation suffix.
  • packages/core/src/job.ts:198-239: the independent Job persists its terminal background result.
  • services/www/src/docs/content/tools.mdx:122-125: background calls are documented to notify the Session when they finish.

The affected Shell, ShellTool, Plugin and Job files are identical between the tested custom HEAD and fixed official 63c1a27. A separate coordinating-audit comparison at upstream dev commit 60a5a6c found only three AI-file changes relative to that official snapshot; these affected sources remained identical. Neither the full official revision nor 60a5a6c was separately executed in this reproduction.

This is distinct from child-session reactivation discussed in #48826: the reproduced failure is cancellation of a shell completion observer during Plugin.activate suffix reconstruction. A preliminary, non-exhaustive duplicate check has not identified an exact match.

Observed once in one controlled run, including the unchanged control. No live configuration watcher, HTTP API, server restart or Location-eviction scenario was exercised. Restart compensation was not tested. The retained launcher was subsequently adjusted to propagate the already-recorded Bun exit code after cleanup; that launcher-only change was not rerun and the test assertions were not changed.

Suggested solution (not implemented)

Own the completion observer at a lifetime matching its Job, or reliably reclaim pending shell notifications after the relevant activation is reconstructed. Preserve generation-aware ownership, stable notification IDs, and admission-before-acknowledgment deduplication. This need not introduce a permanent global subscription or restart unrelated work.

Activity

  1. opencode-agent commented on Oct 18, 2026

    @opencode-agent
    Contributor

    Thanks for the detailed report. I was able to reproduce this on the latest dev (60a5a6c). The relevant source files are unchanged since v2.0.32.

    The script below is a core test that follows your steps:

    1. Activate [audit.prefix (no-op), opencode.tool.shell] through Plugin.activate. The shell plugin is given its services without Scope, as PluginInternal.list does.
    2. Run the shell tool with { "command": "sleep 1; printf fixture-output", "background": true }. It returns running, with one pendingBackground entry.
    3. Re-activate with only [opencode.tool.shell], then wait for the job.

    Results:

    • Rebuild: the shell plugin starts twice and its first scope is closed. The job completes with fixture-output, but the session gets 0 notifications and pendingBackground keeps the completed entry.
    • Control (re-activating the same, unchanged list): 1 notification, and pendingBackground is empty.

    This matches your analysis: notifyWhenDone is started in the plugin's activation scope (forkIn(scope) in tool/plugin/shell.ts), and that scope is closed when the shell plugin is rebuilt.

    Related, but not the same problem:

    Reproduction script (shell-suffix-repro.test.ts)
    import fs from "fs/promises"
    import { realpathSync, watch } from "node:fs"
    import os from "os"
    import path from "path"
    import { describe, expect } from "bun:test"
    import { Cause, Context, Deferred, Duration, Effect, Exit, Fiber, Layer, Queue, Scope, Stream } from "effect"
    import { Money } from "@opencode/schema/money"
    import { AppNodeBuilder } from "@opencode/core/effect/app-node-builder"
    import { LayerNode } from "@opencode/util/effect/layer-node"
    import { makeGlobalNode, makeLocationNode } from "@opencode/util/effect/app-node"
    import { filesystem } from "@opencode/util/effect/app-node-platform"
    import { Database } from "@opencode/core/database/database"
    import { Bus } from "@opencode/core/bus"
    import { Config } from "@opencode/core/config"
    import { Environment } from "@opencode/core/environment/index"
    import { FSUtil } from "@opencode/util/fs-util"
    import { Global } from "@opencode/util/global"
    import { Location } from "@opencode/core/location"
    import { FileAccess } from "@opencode/core/file-access"
    import { LocationServiceMap } from "@opencode/core/location-service-map"
    import { Model } from "@opencode/core/model"
    import { Provider } from "@opencode/core/provider"
    import { AbsolutePath } from "@opencode/core/schema"
    import { Agent } from "@opencode/core/agent"
    import { Job } from "@opencode/core/job"
    import { Session } from "@opencode/core/session"
    import { SessionEvent } from "@opencode/core/session/event"
    import { SessionExecution } from "@opencode/core/session/execution"
    import { SessionMessage } from "@opencode/core/session/message"
    import { SessionStore } from "@opencode/core/session/store"
    import { Permission } from "@opencode/core/permission"
    import { PermissionSaved } from "@opencode/core/permission/saved"
    import { Plugin } from "@opencode/core/plugin"
    import { PluginSupervisor } from "@opencode/core/plugin/supervisor"
    import { Shell } from "@opencode/core/shell"
    import { ShellSelect } from "@opencode/core/shell/select"
    import { ID } from "@opencode/schema/shell"
    import { ShellTool } from "@opencode/core/tool/plugin/shell"
    import { ToolOutput } from "@opencode/core/tool-output"
    import { Tool } from "@opencode/core/tool"
    import { tmpdir, tmpdirScoped } from "./fixture/tmpdir"
    import { tempGlobalLayer } from "./fixture/global"
    import { offlineModels } from "./fixture/models"
    import { testEffect } from "./lib/effect"
    import { permissionLayer } from "./lib/permission"
    import { Expected } from "./lib/session-message"
    import { toolIdentity, executeTool, registerToolPlugin, toolDefinitions } from "./lib/tool"
    
    const sessionID = Session.ID.make("ses_shell_tool_test")
    const sessionModel = Model.Ref.make({ id: Model.ID.make("test"), providerID: Provider.ID.make("test") })
    const assertions: Permission.AssertInput[] = []
    let denyAction: string | undefined
    let afterPermission = (_input: Permission.AssertInput): Effect.Effect<void> => Effect.void
    
    const permission = permissionLayer({
      assert: (input) =>
        Effect.sync(() => assertions.push(input)).pipe(
          Effect.andThen(Effect.suspend(() => afterPermission(input))),
          Effect.andThen(
            input.action === denyAction
              ? Effect.fail(
                  new Permission.BlockedError({
                    rules: [],
                    permission: input.action,
                    resources: input.resources,
                  }),
                )
              : Effect.void,
          ),
        ),
    })
    
    const reset = () => {
      assertions.length = 0
      denyAction = undefined
      afterPermission = () => Effect.void
    }
    
    const executionNode = makeGlobalNode({
      service: SessionExecution.Service,
      layer: Layer.effect(
        SessionExecution.Service,
        Effect.gen(function* () {
          const bus = yield* Bus.Service
          const store = yield* SessionStore.Service
          const complete = Effect.fn("ShellTest.complete")(function* (id: Session.ID) {
            const session = yield* store.get(id)
            if (!session) return
            const assistantMessageID = SessionMessage.ID.create()
            yield* bus.publish(SessionEvent.Step.Started, {
              sessionID: id,
              assistantMessageID,
              agent: session.agent ?? Agent.ID.make("code"),
              model: sessionModel,
              started: 0,
            })
            yield* bus.publish(SessionEvent.Text.Started, {
              sessionID: id,
              assistantMessageID,
              ordinal: 0,
            })
            yield* bus.publish(SessionEvent.Text.Ended, {
              sessionID: id,
              assistantMessageID,
              ordinal: 0,
              text: "ok",
            })
            yield* bus.publish(SessionEvent.Step.Ended, {
              sessionID: id,
              assistantMessageID,
              finish: "stop",
              cost: Money.USD.zero,
              tokens: { input: 0, output: 0, reasoning: 0, cache: { read: 0, write: 0 } },
            })
          })
          return SessionExecution.Service.of({
            active: Effect.succeed(new Set()),
            isActive: () => Effect.succeed(false),
            resume: complete,
            wake: () => Effect.void,
            interrupt: () => Effect.succeed(false),
            awaitIdle: (id) => complete(id).pipe(Effect.exit, Effect.asVoid),
          })
        }),
      ),
      deps: [Bus.node, SessionStore.node],
    })
    
    const shellPluginSupervisor = makeLocationNode({
      name: "test/shell-plugins",
      layer: Layer.effectDiscard(registerToolPlugin(ShellTool.Plugin)),
      deps: [
        Config.node,
        Environment.node,
        FileAccess.node,
        Permission.node,
        Session.node,
        Job.node,
        Shell.node,
        ShellSelect.node,
        Tool.node,
      ],
    })
    
    const nodes = LayerNode.group([
      Database.node,
      Bus.node,
      Job.node,
      Session.node,
      SessionExecution.node,
      LocationServiceMap.node,
      filesystem,
      FSUtil.node,
      Global.node,
    ])
    const replacements = [
      SessionExecution.node.replace(executionNode),
      Permission.node.replace(permission),
      Global.node.replace(tempGlobalLayer),
      offlineModels,
    ] satisfies LayerNode.Replacements
    const productionIt = testEffect(AppNodeBuilder.build(nodes, replacements))
    const it = testEffect(
      AppNodeBuilder.build(nodes, [...replacements, PluginSupervisor.node.replace(shellPluginSupervisor)]),
    )
    const permissionIt = testEffect(
      AppNodeBuilder.build(LayerNode.group([nodes, PermissionSaved.node]), [
        SessionExecution.node.replace(executionNode),
        Global.node.replace(tempGlobalLayer),
        PluginSupervisor.node.replace(shellPluginSupervisor),
        offlineModels,
      ]),
    )
    
    const call = (input: typeof ShellTool.Input.Type, id = "call-shell") => ({
      sessionID,
      ...toolIdentity,
      call: { type: "tool-call" as const, id, name: "shell", input },
    })
    
    const isWindows = process.platform === "win32"
    const cwdCommand = isWindows ? "(Get-Location).Path; Start-Sleep -Milliseconds 100" : "pwd"
    const helloCommand = isWindows ? "[Console]::Out.Write('hello'); Start-Sleep -Milliseconds 100" : "printf hello"
    const stderrCommand = isWindows
      ? "[Console]::Error.Write('stderr only'); Start-Sleep -Milliseconds 100"
      : "printf 'stderr only' >&2"
    const mixedOutputCommand = isWindows
      ? "[Console]::Out.Write('stdout'); Start-Sleep -Milliseconds 50; [Console]::Error.Write('stderr'); Start-Sleep -Milliseconds 100"
      : "printf stdout; sleep 0.05; printf stderr >&2"
    const idleCommand = isWindows ? "Start-Sleep -Seconds 60" : "sleep 60"
    const bodyExitCommand = isWindows
      ? "[Console]::Out.Write('body'); Start-Sleep -Milliseconds 100; exit 7"
      : "printf body && exit 7"
    const overflowCommand = (bytes: number) =>
      isWindows
        ? `[Console]::Out.Write('output-start' + ('x' * ${bytes}) + 'output-end'); Start-Sleep -Milliseconds 100`
        : `printf output-start; head -c ${bytes} /dev/zero | tr '\\0' 'x'; printf output-end`
    const lineOverflowCommand = isWindows
      ? "[Console]::Out.Write('one' + [Environment]::NewLine + 'two' + [Environment]::NewLine + 'three')"
      : "printf 'one\\ntwo\\nthree'"
    const progressOverflowCommand = (bytes: number, release: string) =>
      isWindows
        ? `[Console]::Out.Write(('x' * ${bytes})); while (!(Test-Path -LiteralPath '${release}')) { Start-Sleep -Milliseconds 50 }`
        : `head -c ${bytes} /dev/zero | tr '\\0' 'x'; while [ ! -e '${release}' ]; do sleep 0.05; done`
    
    const withSession = <A, E, R>(directory: string, body: (registry: Tool.Interface) => Effect.Effect<A, E, R>) =>
      Effect.gen(function* () {
        const sessions = yield* Session.Service
        const location = Location.Ref.make({ directory: AbsolutePath.make(directory) })
        yield* sessions.create({
          id: sessionID,
          title: "shell test",
          location,
          model: sessionModel,
        })
        const locations = yield* LocationServiceMap.Service
        const locationLayer = locations.get(location)
        return yield* Effect.gen(function* () {
          const plugins = yield* Plugin.Service
          yield* plugins.awaitActivation
          const registry = yield* Tool.Service
          return yield* body(registry)
        }).pipe(Effect.provide(locationLayer), Effect.ensuring(locations.invalidate(location)))
      })
    
    
    // Repro for anomalyco/opencode#56729
    const scenario = (rebuild: boolean) =>
      Effect.acquireUseRelease(
        Effect.promise(() => tmpdir()),
        (tmp) => {
          reset()
          return withSession(tmp.path, (registry) =>
            Effect.gen(function* () {
              const plugins = yield* Plugin.Service
              const jobs = yield* Job.Service
              const bus = yield* Bus.Service
              const starts = { shell: 0, closed: 0 }
              // Capture services only (no Scope), like PluginInternal.list.
              const context = Context.omit(Scope.Scope)(yield* Effect.context<any>())
              const prefix: Plugin.Generation = { id: "audit.prefix", revision: "1", effect: () => Effect.void }
              const shellGen: Plugin.Generation = {
                id: ShellTool.Plugin.id,
                revision: "1",
                effect: (host) =>
                  Effect.gen(function* () {
                    starts.shell++
                    yield* Effect.addFinalizer(() => Effect.sync(() => starts.closed++))
                    yield* ShellTool.Plugin.effect(host as any)
                  }).pipe(Effect.provide(context)) as any,
              }
              yield* plugins.activate([prefix, shellGen])
              const notices: unknown[] = []
              yield* bus.subscribe(SessionEvent.InboxEnqueued).pipe(
                Stream.filter((e) => e.data.sessionID === sessionID && e.data.item.type === "synthetic"),
                Stream.runForEach((e) => Effect.sync(() => notices.push(e.data.item.payload))),
                Effect.forkScoped({ startImmediately: true }),
              )
              const settled = yield* executeTool(
                registry,
                call({ command: "sleep 1; printf fixture-output", background: true }, "call-bg-" + rebuild),
              )
              const shellID = settled.metadata?.shellID as string
              expect(settled.output?.status).toBe("running")
              expect((yield* jobs.pendingBackground).length).toBe(1)
    
              yield* plugins.activate(rebuild ? [shellGen] : [prefix, shellGen])
    
              const result = yield* jobs.wait({ id: shellID })
              yield* Effect.promise(() => Bun.sleep(1500))
              const pending = yield* jobs.pendingBackground
              console.log(
                JSON.stringify({ rebuild, starts, job: result.info?.status, notices: notices.length, pending }, null, 1),
              )
              expect(result.info?.status).toBe("completed")
              expect(notices.length).toBe(1)
              expect(pending.length).toBe(0)
            }),
          )
        },
        (tmp) => Effect.promise(() => tmp[Symbol.asyncDispose]().then(() => undefined)),
      )
    
    productionIt.live("control: unchanged activation delivers the background notification", () => scenario(false), {
      timeout: 20_000,
    })
    productionIt.live("rebuilt shell suffix still delivers the background notification", () => scenario(true), {
      timeout: 20_000,
    })

    Run it: bun shell-suffix-repro.test.ts

  2. reynold-hu commented on Oct 18, 2026

    @reynold-hu
    Contributor

    I'm working on this and will open a PR. The plan matches the suggested direction: the ShellTool plugin reclaims pending shell backgrounds when it is activated, waits for still-running jobs and replays the notification from the persisted Job marker for finished ones, then acknowledges the marker. That keeps the observer tied to the current activation without moving notification ownership out of the shell tool.

  3. reynold-hu commented on Oct 18, 2026

    @reynold-hu
    Contributor

    Opened #56780 with the reclaim approach. The new repro test fails on the parent commit (notices: 0, pending: 1) and passes with the fix; the unchanged-activation control is covered in the same file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions