V2: first Location can expose an empty plugin generation #35556

Activity

1. kitlangton

   added

   bugSomething isn't workingSomething isn't working

   coreAnything pertaining to core functionality of the application (opencode server stuff)Anything pertaining to core functionality of the application (opencode server stuff)

   2.0

   on Jul 6, 2026
2. kitlangton commented on Jul 6, 2026

   

   kitlangton

   on Jul 6, 2026

   ContributorAuthor

   More actions

   Updated the issue body with new findings verified on origin/v2 at 175d200.

   The key addition: losing the registration/boot race is not just a delayed activation, it is permanent. The supervisor's only reload triggers are the forked boot reload and config Event.Updated, SdkPlugins.register publishes nothing, and the Location cache never expires under active traffic. I reproduced this deterministically at two levels: a core test that polls for 2 seconds after post-boot registration (never appears, then activates immediately after one manually published config event), and an embedded SDK test where the already-booted Location never sees the plugin while a Location booted after registration does.

   This means the fix needs both halves: gating Location readiness on initial activation (this issue) and making post-boot registration either apply or fail loudly (#34799).
3. kitlangton commented on Jul 6, 2026

   

   kitlangton

   on Jul 6, 2026

   ContributorAuthor

   More actions

   Narrowed this issue to the remaining initial-generation race. The permanent post-boot registration miss is fixed by #35576; PluginSupervisor initial readiness remains intentionally unchanged and is the only behavior tracked here now.
4. kitlangton commented on Jul 7, 2026

   

   kitlangton

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Production evidence from the personal OpenCode Agent on V2 commit f2f62438b confirms that this race can recur well after process startup because the cached Location is evicted and later rebuilt.

   Timeline (UTC)

   • 2026-07-06 23:50:01.863: Location services booted for the agent root.
   • 2026-07-07 02:21: last activity before the idle period.
   • 2026-07-07 03:21:35.971: the Location watcher stopped, matching the configured 60-minute LayerMap idle TTL.
   • 2026-07-07 10:47:04.808: the same Location booted again on the next Telegram message.
   • 2026-07-07 10:47:07.085: the first Session drain failed with SessionRunnerModel.ModelUnavailableError: Model unavailable: console-openai/gpt-5.6-sol.
   • 2026-07-07 10:47:08.201: agent.updated was published.
   • 2026-07-07 10:47:08.450: catalog.updated was published.

   The model lookup therefore ran about 1.36 seconds before the catalog generation finished.

   Second symptom: advertised tools failed closed

   At 11:00 the same Session was able to call the model, and the model was advertised read, glob, shell, and websearch. Every call failed locally (provider.executed: false) with the generic wrappers:

   • Unable to read HEARTBEAT.md
   • Unable to find files matching HEARTBEAT.md
   • Unable to execute command: ...
   • Unable to search the web for test

   These tools all pass through PermissionV2.assert. If the configured SDK agent is absent, PermissionV2.configured returns missingAgentPermissions (deny all). Meanwhile SessionRunnerLLM passes agent.info?.permissions to tool materialization, so an absent agent supplies undefined: tools are advertised, then denied when executed. The tool-specific mapError wrappers erase the underlying denial and make the failures look unrelated.

   Restarting the service rebuilt the Location after plugin registration and immediately restored the model, agent permissions, and tools.

   Deterministic same-Location reproduction

   I added a focused test at packages/core/test/location-layer.test.ts:

   1. Register an SDK plugin that contributes an allow-all agent.
   2. Open a Location, await the current supervisor's ready, and verify the agent exists.
   3. Invalidate the same cached Location (the deterministic equivalent of idle eviction).
   4. Gate the plugin's second activation on a Deferred.
   5. Reopen the same Location and read AgentV2 after activation has started but before it is released.
   6. Expect the SDK agent to exist.

   On f2f62438b, the test deterministically fails:

   LocationServiceMap > waits for SDK agents when a cached Location is reopened
   Expected: defined
   Received: undefined
   

   bun typecheck passes; only the readiness assertion fails.

   This expands the impact boundary: "first Location" includes the first request after any idle eviction, not just initial process startup. The fix needs a readiness contract that Session execution can rely on for every fresh Location generation. The current PluginSupervisor.Service.ready is only a join of its boot reload fiber; whether that primitive is sufficient or needs a stronger generation-aware contract should be decided explicitly rather than assuming that merely having a ready field closes the race.
5. kitlangton commented on Jul 7, 2026

   

   kitlangton

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Follow-up: the SDK plugin is not lost across Location eviction. SdkPlugins.Store is host-global, and the reopened Location does start the plugin's second activation. The problem is that Location-scoped contributions (agents, tools, catalog entries, hooks) are torn down on eviction and then repopulated asynchronously.

   More importantly, simply awaiting the current PluginSupervisor.Service.ready from Session execution is not a complete fix for hosts that register plugins later.

   I added a second deterministic test covering registration during reopen:

   1. Boot a Location with SDK plugin A and await ready.
   2. Invalidate and reopen the same Location.
   3. Block A's activation during the reopened initial generation.
   4. Register SDK plugin B while that initial generation is blocked.
   5. Call supervisor.ready after B's registration returns.
   6. Release A, let the SDK update reload start, and block B's activation.
   7. Assert that ready has not completed while B is still activating.

   Current behavior fails the assertion: ready has already completed even though B's activation is blocked. This is expected from the implementation: ready is Fiber.join(fiber) for the one boot reload; it does not represent subsequent reloads queued by sdk.plugin.updated.

   So there are two distinct guarantees to design:

   1. Fresh-Location readiness: no Session execution may observe the empty/partial initial generation.
   2. Registration ordering: if sdk.register(B) completes before a Session asks for readiness, that Session must not run against a generation older than B.

   A fixed boot-fiber join can provide (1), but not (2). A stronger contract likely needs monotonically increasing desired/applied plugin revisions (or an equivalent generation token): registration increments the host SDK revision; each Location supervisor records the revision included in an activated generation; Session execution captures the current desired revision and waits until that Location has applied at least that revision. This gives a clear happens-before rule without making registration synchronously wait for every booted Location.

   Config/plugin-file updates may need the same desired/applied-generation treatment if they require equivalent ordering guarantees. Otherwise the contract should explicitly distinguish SDK registration consistency from eventually consistent config reloads.
6. kitlangton commented on Jul 7, 2026

   

   kitlangton

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Confirmed a second, persistent post-eviction bug with a deterministic test using the same replacement graph as the embedded SDK host.

   OpenCode.create() owns one shared SdkPlugins.Store. createEmbeddedRoutes(store) passes SdkPlugins.layerWithStore(store) as an AppNode replacement. When the lazy Location graph hoists that replacement, the replacement layer is owned by the Location scope. But layerWithStore installs this finalizer:

   yield* Effect.addFinalizer(() =>
     Effect.sync(() => {
       store.plugins.clear()
     }),
   )

   Therefore evicting any Location closes that Location's SdkPlugins service and clears the process-global shared store. On reopen, the new PluginSupervisor can complete its initial reload normally, but sdk.all() is empty. SDK-contributed agents and tools do not return at all; this is not merely the initial readiness race. A process restart makes the host register them again, explaining why restart repaired both the persona permissions and tools.

   The focused test:

   1. Creates the production-shaped Location map with [SdkPlugins.node, SdkPlugins.layerWithStore(sharedStore)] as a replacement.
   2. Inserts an SDK plugin contributing an agent.
   3. Boots the Location, awaits PluginSupervisor.ready, and verifies the agent exists.
   4. Calls LocationServiceMap.invalidate(ref), the deterministic equivalent of idle eviction.
   5. Reopens the same Location, awaits the new supervisor's ready, and expects the agent to exist.

   It fails deterministically after ready:

   LocationServiceMap > preserves embedded SDK plugins after Location eviction
   Expected: defined
   Received: undefined
   

   bun typecheck passes.

   So there are two independent defects:

   1. Lifetime ownership: a Location-scoped finalizer clears the host-global SDK store, permanently dropping late/startup SDK plugins after eviction.
   2. Readiness ordering: while a generation is activating, Session execution can observe partial Location state; the current fixed boot-fiber ready also does not cover a later reload queued during boot.

   The lifetime fix should make the owner of the shared host store solely responsible for clearing it. A Location-scoped SdkPlugins service may reference the store but must never clear it. The readiness/generation fix remains separately necessary.
7. kitlangton

   self-assigned this

   on Jul 7, 2026
8. kitlangton commented on Jul 7, 2026

   

   kitlangton

   on Jul 7, 2026

   Last edited by kitlangton

   ContributorAuthor

   More actions

   Added a deterministic reproduction on latest v2 (8deb0e5780). It covers both boundaries:

   • While initial plugin activation is blocked, AgentV2.select("explore") returns { id: "explore", info: undefined }, and tool materialization advertises all built-ins, including shell.
   • A Session whose selected agent is unavailable sends an actual LLM request containing shell.

   The focused tests fail with shell present in both tool lists, while bun typecheck passes. Draft PR: #35755
9. kitlangton commented on Jul 7, 2026

   

   kitlangton

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Proposed readiness semantics

   The supervisor should expose a drain wait for plugin activation. When Session execution waits, the supervisor must finish the current generation and also apply any plugin/config registrations requested while that generation was activating. It returns only when requested === applied at the completion boundary. A registration after that boundary belongs to the next synchronization.

   requested = 1
   start generation 1
   plugin B registers -> requested = 2
   finish generation 1
   activate generation 2
   finish generation 2
   requested === applied === 2 -> return
   

   This is not a promise that the whole process is globally quiet. Readiness means:

   • every included plugin module was imported
   • every included plugin's top-level setup returned
   • awaited agent, model, command, skill, tool, and hook contributions were registered
   • the activation batch committed its state projections
   • final config projection plugins ran
   • no tracked plugin reload was pending when the wait completed

   Readiness does not mean:

   • arbitrary background fibers completed
   • MCP servers connected
   • ongoing listeners terminated
   • future plugin registrations cannot occur
   • network refreshes forked by plugins completed

   This implies a plugin contract: top-level setup must await every initial contribution required for correct Session execution. It may fork only ongoing listeners and later refresh work. For example, an initial provider refresh needed to construct the catalog must be awaited; the listener for future credential changes should be forked.

   Session execution should call this wait before resolving its agent, model, instructions, or tools. After the wait, an explicitly selected missing agent fails with a typed AgentNotFoundError; it never proceeds with permissions: undefined or falls back to another agent.

   Location acquisition and TUI bootstrap remain progressive. They do not wait. This keeps first paint fast while giving model execution the stronger consistency contract it requires.

   Draft reproducer: #35755.
10. opencode-agent commented on Jul 8, 2026

    

    opencode-agentBot

    on Jul 8, 2026 – with OpenCode Agent

    Contributor

    More actions

    Audit update (2026-07-08): #35725 fixed the separate SDK-plugin lifetime/eviction defect and merged into v2. The transient initial-generation readiness race described here remains open; #35755 is the active deterministic reproduction, while #35660 is a dev-targeted proposed wait-for-boot fix.
11. kitlangton commented on Jul 8, 2026

    

    kitlangton

    on Jul 8, 2026

    ContributorAuthor

    More actions

    PR #35755 now implements the final one-shot readiness contract from #35942 at f49e25408b. Session execution waits for one successfully activated, quiescent startup generation, then fails with typed AgentNotFoundError before any model request if the selected agent is still absent. Later hot reload remains asynchronous.

    The final simplify pass bounds pending invalidations with capacity-one sliding queues because each activation rereads current Config and SDK state. Combined location, runner, recorded, and provider suites pass (343 tests), and repository typecheck passes (31 tasks). Linux CI is green; Windows CI is still running.
12. kitlangton commented on Jul 8, 2026

    

    kitlangton

    on Jul 8, 2026

    ContributorAuthor

    More actions

    Fixed by merged PR #35755 (5b39972947). The initial Location plugin generation is now gated at Session execution, startup updates settle before the one-shot barrier opens, and missing selected agents fail before model execution.
13. kitlangton

    closed this as completedon Jul 8, 2026
14. Fade78 commented on Oct 11, 2026

    

    Fade78

    on Oct 11, 2026

    More actions

    Same thing here in 2.0.26. Loading agents fail:
    opencode service restart
    opencode debug agents -> []
    (few seconds later)
    opencode debug agents -> [ full list ]

    opencode --standalone can't have agents.