Skip to content

V2: failed location boot is cached for the full 60 minute idle TTL with no recovery #38405

Closed as not planned

Description

@dondetir

Description

On the v2 branch, when a location's service graph fails to boot once (for example a transient plugin, MCP spawn, or git error), the failure is cached for the entire 60 minute idleTimeToLive of the location map. Every later request for that location replays the same stale error, even after the underlying cause is fixed. The only recovery is the debug invalidate endpoint or a server restart.

Cause: buildLocationServiceMap (packages/core/src/location-services.ts) uses LayerMap.make with idleTimeToLive: "60 minutes". In effect 4.0.0-beta.83, RcMap stores the lookup result (success or failure) in the entry's deferred, and MemoMap sets entry.effect = exit on failure, so a failed build is cached exactly like a successful one. Nothing in production ever calls locations.invalidate; the only callers are tests and packages/server/src/handlers/debug.ts.

Minimal repro against the same effect version, showing the failure is still served after the cause is fixed:

import { Context, Duration, Effect, Layer, LayerMap } from "effect"

class Svc extends Context.Service<Svc, { n: number }>()("Svc") {}

let builds = 0
let fail = true
const lookup = (_key: string) =>
  Layer.effect(
    Svc,
    Effect.suspend(() => {
      builds++
      return fail ? Effect.fail(new Error(`boom #${builds}`)) : Effect.succeed({ n: builds })
    }),
  ).pipe(Layer.fresh)

const main = Effect.gen(function* () {
  const map = yield* LayerMap.make(lookup, { idleTimeToLive: Duration.minutes(60) })
  for (let i = 1; i <= 3; i++) {
    const r = yield* map.contextEffect("loc").pipe(Effect.scoped, Effect.exit)
    console.log(`get#${i} failing: builds=${builds} result=${r._tag}`)
  }
  fail = false
  for (let i = 4; i <= 6; i++) {
    const r = yield* map.contextEffect("loc").pipe(Effect.scoped, Effect.exit)
    console.log(`get#${i} fixed: builds=${builds} result=${r._tag}`)
  }
}).pipe(Effect.scoped)

Effect.runPromise(main)

Output: all six gets report builds=1 result=Failure. The build is never retried, and the gets after the fix still fail.

Found while investigating #36677. This is a different bug than the allocation loop reported there, but it lives in the same location lifecycle.

I have a small fix ready (evict the cached entry on boot failure, rate limited so a persistently failing location cannot turn into a hot rebuild loop) and will open a PR.

Plugins

None relevant, code level bug.

OpenCode version

v2 branch, effect 4.0.0-beta.83.

Steps to reproduce

  1. Run a v2 server with a location whose boot fails once (any transient error in a location scoped service).
  2. Fix the underlying cause.
  3. Request the location again within 60 minutes: the request keeps failing with the original cached error.

Activity

  1. github-actions commented on Sep 21, 2026

    @github-actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions