memos-local-plugin: large merged episodes can trigger L2/L3/skill-evolution storm and stall OpenClaw sessions

Summary

During long-running OpenClaw development workflows, memos-local-plugin can create or reopen a very large episode and then trigger a cascade of expensive post-processing (capture / reward rescore / L2 induction / L3 abstraction / skill crystallization). In our case this caused foreground conversations and unrelated sessions to stall together with background memory processing.

This seems typical for real OpenClaw usage: a user may work on a development task for a long time, continue the same topic hours later or the next day, and generate many tool calls. If follow-up turns are merged into the same episode, the resulting episode can become large enough that startup recovery or episode finalization overwhelms the gateway.

Environment

Plugin: @memtensor/memos-local-plugin
Version observed: 2.0.4
Adapter/runtime: OpenClaw local plugin
Memory mode when issue happened: full memory self-evolution (lightweightMemory.enabled=false)
LLM provider: OpenAI-compatible endpoint
Workload type: long-running development task with many tool calls, followed by later related follow-up work

What happened

A long development task produced a large episode. Later follow-up work was classified/merged as follow_up, reopening the same episode. After a gateway restart / plugin reload, the plugin detected the episode as needing recovery/rescore and emitted it as finalized again. That triggered a large background processing cascade.

Observed chain:

A long development task generated hundreds of traces/tool call records.
A later related follow-up turn was merged into the previous episode (reopenReason: follow_up).
Startup recovery marked the closed episode for rescore (recoveryReason: dirty_reward_rescore).
The recovered/finalized episode triggered:
- capture.reflection
- reward processing
- L2 induction
- L3 abstraction
- skill verification / crystallization
Foreground OpenClaw sessions then experienced hook timeouts and stalled runs.

The issue was not caused by the user's foreground task itself. The foreground task merely happened to run while the plugin was processing the large recovered episode.

Evidence from local logs / DB

Example large episode characteristics from the incident:

Episode duration: about 15:10–16:34 local time
Reopened at: about 16:29 local time
reopenReason: follow_up
recoveryReason: dirty_reward_rescore
Traces in DB: 621
Tool calls in DB: 614
Session transcript lines: 237
One turn alone had hundreds of traces/tool calls

Relevant symptoms in gateway logs:

[core.capture.reflection] synth.failed ...
[core.memory.l2] run.start ...
[core.memory.l3] run.start ...
[core.skill] skill.run.start ...
[plugins] [hooks] before_prompt_build handler from memos-local-plugin failed: timed out after 15000ms
[plugins] [hooks] agent_end handler from memos-local-plugin failed: timed out after 30000ms
[diagnostic] stalled session ... reason=active_work_without_progress classification=stalled_agent_run

We also saw L3 JSON/schema failures during this period, for example:

[llm.json] malformed op="l3.abstraction.v2" ...
[core.memory.l3.abstract] abstract.llm_failed ... err="l3.abstraction: 'title' must be a non-empty string"

Those model/schema failures made the cascade more expensive, but the core operational problem appears to be that heavy background memory evolution was not isolated from foreground hooks/sessions.

Expected behavior

Long development workflows should not be able to create a single episode whose recovery or finalization can block or stall unrelated foreground conversations.

In particular:

Memory self-evolution should not block before_prompt_build or agent_end beyond a small budget.
Startup recovery should not immediately trigger unbounded L2/L3/skill evolution for very large episodes.
Follow-up merging should have safeguards for long-running development tasks.
A single large episode should be processed incrementally or deferred, not all at once on gateway reload.

Actual behavior

A large recovered episode triggered heavy processing and foreground sessions started timing out/stalling. The practical effect was that OpenClaw appeared globally stuck even though the foreground task was not itself computationally heavy.

Why this may be common

This is not an edge case for coding agents. A typical user flow is:

Day 1: ask the agent to implement or refactor a feature; many tool calls are produced.
Later the same day or Day 2: ask follow-up questions or request additional changes in the same domain.
The memory plugin treats those messages as follow-ups and merges them into the previous episode.
The episode keeps growing across long-running work and eventually becomes expensive to finalize/recover/evolve.

For users who rely on OpenClaw for long development sessions, this pattern can happen frequently.

Temporary mitigation used

We restored usability by enabling:

algorithm:
  lightweightMemory:
    enabled: true

This prevented the foreground sessions from stalling, but it also reduces or disables much of the advanced L2/L3/skill evolution value.

Suggested improvements

Some possible fixes/design changes:

Episode size limits / hard cutoffs
- Force a new episode after N traces, N tool calls, N characters, or M minutes.
- For example: split after 50 traces / 100 tool calls / 30 minutes, configurable.
Safer follow-up merging
- Do not reopen a closed episode if it is already large or old.
- Add max merge window and max episode size constraints.
- Prefer episode_per_turn or bounded merge behavior for tool-heavy coding sessions.
Background evolution isolation
- L2/L3/skill evolution should run in a separate queue/worker or maintenance window.
- Foreground hooks should have strict time budgets and should never wait for large evolution work.
Scheduled evolution window
- Allow config such as: run L2/L3/skill evolution only during 02:00–05:00.
- During daytime, only record traces/embeddings/retrieval.
Per-episode processing budget
- Limit max traces/tool calls processed per run.
- If budget is exceeded, checkpoint progress and continue later.
Startup recovery guardrails
- Startup recovery should not immediately replay/finalize very large episodes into the full evolution pipeline.
- It should enqueue them with low priority or require an explicit maintenance job.
Better observability
- Log episode size and queue size before launching recovery/evolution.
- Emit warnings when an episode exceeds safe thresholds.

Additional note

The lightweightMemory switch is a useful safety valve, but it is too coarse. It would be better to keep normal trace/embedding/retrieval online while deferring or rate-limiting only the expensive self-evolution stages.

Detailed Root Cause Analysis

After auditing the full plugin source (v2.0.4), I can confirm this is a real and architectural issue affecting both OpenClaw and Hermes Agent identically. Both adapters share the same core pipeline via bridge.{js,cts}.

Trigger Path

agent_end → onTurnEnd → openEpisodeIfNeeded → relation.classify (LLM)
  → finalizeEpisode → runReflect → capture.runReflect (LLM)
    → reward rescore (LLM) → L2 induction (LLM) → L3 abstraction (LLM)
      → skill crystallization (LLM)

Four Concrete Defects

1. No episode size cap — unbounded turn accumulation

There is zero logic for maxTurns, maxEpisodeSize, or episode splitting anywhere in the codebase. When relation.classify consistently returns revision/follow_up (common in long dev sessions), a single episode can accumulate dozens or hundreds of turns before the topic finally switches. The entire accumulated episode then hits the full L2→L3→skill chain at once.

Key locations:

orchestrator.js:openEpisodeIfNeeded() — no turn-count guard
episode-manager.js — addTurn() has no size check

2. `before_prompt_build` synchronously awaits finalize → runReflect

When a topic change is detected in onTurnStart (called from the before_prompt_build hook), the orchestrator does:

const decision = await session.relation.classify({...}); // LLM call
// ...
session.sessionManager.finalizeEpisode(currentEpId);  // triggers runReflect synchronously

This blocks the foreground session's prompt construction while the entire capture→reward→L2→L3 chain runs. A large episode makes this timeout-length.

3. `flush()` in shutdown synchronously drains full chain

await subs.subscriptions.capture.drain();  // LLM calls
await subs.subscriptions.reward.drain();    // LLM calls
await subs.l2.drain();                      // LLM calls ~5s/batch
await subs.l3.drain();                      // LLM calls
await subs.skills.flush();                  // LLM calls

Each .drain() awaits all inflight promises. With a large episode, this chains multiple LLM calls sequentially, blocking the Gateway event loop.

4. No concurrency limit on LLM-heavy background work

The capture subscriber fires void run(...) promises without any semaphore:

const p = runner.runReflect({ episode, closedBy })
    .catch(err => { ... })

Multiple topic-end events can trigger unlimited parallel LLM calls, starving the Node.js event loop and causing WebSocket timeouts in foreground sessions.

Hermes Agent Impact

Identical. Hermes uses bridge.cts which connects to the same core/pipeline/orchestrator.js. The onTurnEnd → finalizeEpisode → runReflect chain is shared 100%. The mitigation (disabling L2/L3/skill-evolution) works on both sides equally.

Proposed Fixes

I plan to implement fixes #1–3 in a single PR:

Episode turn hard limit — auto-finalize when turns.length >= MAX_TURNS (configurable, default ~30)
Async topic-end processing — before_prompt_build only does routing; finalizeEpisode + runReflect moves to a background queue with backpressure
LLM concurrency semaphore — global signal (≤2 concurrent) for capture/reward/L2/L3 LLM calls

Additional real-world case: batch text extraction workload also triggers the same self-evolution storm.

Environment:

OpenClaw 2026.6.6
@memtensor/memos-local-plugin 2.0.6
OpenClaw adapter
lightweightMemory.enabled=false
Workload: batch file/text extraction, many subagent/tool-heavy subtasks, each producing short completion reports

Observed over ~12h:

system_model_status: 5884
policy_evolve: 1096
skill_generate: 511, 510 failed
policy_generate: 154
world_model_generate: 16, all failed
memory_add: 223
memos_search: only 50

So the cost was not from memory retrieval. It came from the full self-evolution chain treating mechanical batch extraction steps as reusable experience and trying to run reward/L2/L3/skill crystallization.

This is slightly different from a single huge development episode: the workload is intentionally repetitive and tool-heavy, but most steps are operational noise and should not enter expensive self-evolution.

Expected:

Batch/tool-heavy extraction tasks should either auto-downgrade to lightweight capture, or be marked as no_evolution.
The user should be able to choose per session / per episode:
- full self-evolution
- lightweight memory only
- no memory capture
Expensive stages should have budgets/quotas:
- max LLM calls per episode
- max skill_generate per hour
- max policy/world model evolution per time window
Mechanical repeated episodes should be deduplicated before L2/L3/skill stages.

Temporary mitigation:

algorithm:
  lightweightMemory:
    enabled: true

But this is too coarse because it disables advanced memory value globally. A per-session or per-workload switch would be much better for OpenClaw users.

fix: guard against episode storm stalling foreground sessions (#1844)

release: merge dev-v2.0.25 into main (#2156)

Fixed by #1844 (guard against episode storm stalling foreground search), merged into dev-v2.0.25 and released to main via #2156 (v2.0.25). Closing.

release: merge dev-v2.0.25 into main (MemTensor#2156)

Navigation Menu

Repository navigation

memos-local-plugin: large merged episodes can trigger L2/L3/skill-evolution storm and stall OpenClaw sessions #1755

Summary

Environment

What happened

Evidence from local logs / DB

Expected behavior

Actual behavior

Why this may be common

Temporary mitigation used

Suggested improvements

Additional note

Detailed Root Cause Analysis

Trigger Path

Four Concrete Defects

1. No episode size cap — unbounded turn accumulation

2. `before_prompt_build` synchronously awaits finalize → runReflect

3. `flush()` in shutdown synchronously drains full chain

4. No concurrency limit on LLM-heavy background work

Hermes Agent Impact

Proposed Fixes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

memos-local-plugin: large merged episodes can trigger L2/L3/skill-evolution storm and stall OpenClaw sessions #1755

Description

Summary

Environment

What happened

Evidence from local logs / DB

Expected behavior

Actual behavior

Why this may be common

Temporary mitigation used

Suggested improvements

Additional note

Activity

0xDe1ty commented on May 31, 2026

Detailed Root Cause Analysis

Trigger Path

Four Concrete Defects

1. No episode size cap — unbounded turn accumulation

2. before_prompt_build synchronously awaits finalize → runReflect

3. flush() in shutdown synchronously drains full chain

4. No concurrency limit on LLM-heavy background work

Hermes Agent Impact

Proposed Fixes

MaiHHConnect commented on May 31, 2026

adamj0909-rgb commented on Jun 26, 2026

Memtensor-AI commented on Jul 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. `before_prompt_build` synchronously awaits finalize → runReflect

3. `flush()` in shutdown synchronously drains full chain