Repository navigation
bug: ghost trace in episode.trace_ids_json causes infinite rescore loop (590 calls, 14.5 RMB wasted) #1966
Description
Summary
A ghost trace ID () was added to an episode's but never existed in the table. This caused to return indefinitely, triggering 590 reward rescore calls over 5 days and wasting ~14.5 RMB on LLM calls.
Root Cause
1. Ghost trace creation
The episode was reopened with . During the follow-up merge, was called which directly overwrites without validating that trace IDs actually exist in the table.
// episodes.js line 80-82
appendTrace(id, traceIds) {
appendTrace.run({ id, trace_ids_json: toJsonText(traceIds) });
// No validation that traceIds exist in traces table!
}2. Reward traceCount mismatch
loads traces via which only returns existing traces:
// reward.js line 48-52
const traces = traceIds.length > 0
? deps.tracesRepo.getManyByIds(traceIds).sort(...)
: [];This returns 11 traces (ghost excluded). After scoring: reward.traceCount = 11.
3. Dirty check compares against episode.traceIds.length
// memory-core.js line 1003-1006
const traceCount = reward.traceCount;
if (typeof traceCount === number) {
return traceCount !== (ep.traceIds?.length ?? 0);
// 11 !== 12 → true → dirty forever!
}4. Infinite loop
runs every 10 minutes, finds the episode dirty, rescores, gets 11 vs 12 mismatch, episode stays dirty. Repeat 590 times.
Evidence
Episode: ep_95n61b3jzycd
trace_ids_json count: 12 (including ghost tr_xhbp6c9p450r)
tr_xhbp6c9p450r in traces table: 0 (does not exist)
reward.traceCount: 11
episodeRewardIsDirty: true (traceCount mismatch)
Reward calls: 590 over 5 days (06-18 to 06-22)
Suggested Fixes
- appendTrace validation: Validate trace IDs exist before appending
- Dirty check resilience: Compare against actual existing trace count, not episode.traceIds.length
- Rescore retry limit: Add max retry count per episode to prevent infinite loops
- Ghost trace cleanup: Add startup scan to remove orphaned trace IDs from episodes
Related Issues
- memos-local-plugin: large merged episodes can trigger L2/L3/skill-evolution storm and stall OpenClaw sessions #1755 — large merged episodes can trigger L2/L3/skill-evolution storm (same infinite rescore pattern)
Environment
- MemOS: v2.0.20
- Agent: hermes
- OS: Windows 11
Activity
Memtensor-AI commentedon Jun 30, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
35fc16a4a540b16aWorking branch:
feature/autodev-1966Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(memos-local-plugin): tolerate ghost trace IDs in reward dirty che…
Memtensor-AI commentedon Jun 30, 2026
✅ AutoDev task
35fc16a4a540b16acompleted.Summary: Fix #1966 — the reward dirty-check infinite loop triggered by ghost trace IDs lingering in
episodes.trace_ids_json(the reporter wasted 590 LLM calls / ~14.5 RMB over 5 days). The bug lives inapps/memos-local-plugin:episodeRewardIsDirtycomparedreward.traceCountagainst the rawep.traceIds.length, so a single orphan ID kept the episode dirty forever andrunDirtyClosedRewardScanre-scored it every 10 minutes.Three-layer fix (defence in depth): (A)
episodeRewardIsDirtynow comparesreward.traceCountagainst the count of trace IDs that actually exist in thetracestable, via a newtraces.countExistinghelper (singleSELECT COUNT(*)per 900-id chunk, no BLOB hydration — same cost profile as thehasAnyNewerThanfix added for #1787). This deterministically breaks the loop. (B)repos.episodes.appendTracenow strips ghost IDs before writingtrace_ids_json, so new ghosts cannot accumulate. (C) Two new traces-repo helpers —countExisting(ids)andfilterExistingIds(ids)— back both fixes with the same chunked-SELECT pattern. The issue's other two suggested fixes (retry counter, startup ghost-scan) were intentionally skipped: Fix A breaks the loop without them, and adding either would slow init or introduce extra state to reason about.Tests:
tests/unit/storage/repos.test.tsupdated to match the newappendTracecontract (insert real traces first, then assert ghost-strip on mixed / all-ghost / re-append inputs) and gains coverage for both new helpers;tests/unit/pipeline/memory-core.test.tsadds an integration test "does not rescore a closed episode whose only mismatch is a ghost trace ID (#1966)" that seeds the exact reporter scenario and assertsr_taskandmeta.reward.rHumanstay untouched afterinit().npm run lintpasses clean;npx vitest run tests/unit→ 1041 passed, only 2 pre-existing failures remain (migrator schema-column drift and a traces-count pagination cap, both confirmed to fail ondev-20260624-v2.0.22viagit stashand both unrelated to ghost-trace handling). No DB migrations, no schema changes, no API surface changes; all 4 existingappendTracecallers insert trace rows before referencing them, so the new validation is a no-op for valid inputs.Base branch:
mainBranch:
feature/autodev-1966Commit:
ed6d25fPR: #2004
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller
Fix #1966: bug: ghost trace in episode.trace_ids_json causes infinite…
Fix MemTensor#1966: bug: ghost trace in episode.trace_ids_json causes…