local-plugin v2.0.8: gateway CPU 100% — synchronous full-table vector scan (scanAndTopK) + unbounded trace re-insertion from dedup pagination cap #2076

Summary

Running the OpenClaw local memory plugin (@memtensor/memos-local-plugin v2.0.8), the gateway node process pinned one CPU core at 100% for 40+ minutes on startup, holding 4.2 GB RSS. Investigation traced this to two distinct bugs in the plugin:

1. Vector retrieval does a synchronous O(n) full-table scan on the Node main thread, decoding every vector BLOB and computing cosine in-process — this starves the event loop (100% CPU, GC/IO threads idle).
2. A pagination cap in the capture-side dedup read causes the same trace to be re-inserted indefinitely, bloating the traces table (in our case 518,375 rows of which 437,792 — 84% — were exact duplicates), which is what made bug feat: add internet retrieval and CoT functionality to MemOS #1 catastrophic.

Together: bug #2 grows the table without bound → bug #1 loads the whole thing synchronously on every dreaming/retrieval cycle → CPU melts.

Environment

• @memtensor/memos-local-plugin v2.0.8 (MemOS Local Memory (V7))
• Node 26, better-sqlite3
• dreaming.enabled: true, embedding.enabled: false
• memos.db had grown to 6.8 GB (traces table alone = 5.96 GB, ~4.2 GB of that in vec_summary/vec_action BLOBs)

Evidence — native stack of the 100% CPU main thread (gdb)

#0  pread64
#1  unixRead                         (better_sqlite3.node)
#2  accessPayload                    (better_sqlite3.node)
#3  vdbeColumnFromOverflow           (better_sqlite3.node)
#4  sqlite3VdbeExec
#5  sqlite3_step
#6  Statement::JS_all(...)           <-- synchronous .all()
#7  Builtins_CallApiCallbackOptimizedNoProfiling  (libnode)

strace -c over 3s on the main thread: 56,637 pread64 calls, 99.97% of time — i.e. the thread is doing nothing but synchronously reading BLOB overflow pages off a multi-GB table.

Bug #1 — scanAndTopK blocks the event loop

dist/core/storage/vector.js:149

export function scanAndTopK(db, table, selectExtra, query, k, opts) {
    const { vecColumn, norm2Column, where, params, hardCap } = opts;
    const cols = ["id", vecColumn, ...(norm2Column ? [norm2Column] : []), ...selectExtra];
    const sql = [
        `SELECT ${cols.join(", ")} FROM ${table}`,
        where ? `WHERE ${where}` : "",
        `LIMIT ${hardCap ?? 100000}`,          // default cap = 100,000 rows
    ].filter(Boolean).join(" ");
    const rows = db.prepare(sql).all(params);  // <-- loads up to 100k rows w/ vector BLOBs synchronously
    const decoded = [];
    for (const r of rows) {
        const vec = decodeVector(r[vecColumn]); // <-- decode every BLOB on main thread
        ...
    }
    return topKCosine(query, decoded, k);       // <-- cosine over all of them on main thread
}

Called from dist/core/storage/repos/traces.js:255 (searchByVector).

Problems:

• No ANN/vector index — retrieval is a brute-force O(n) scan over the entire table.
• .all() materializes up to hardCap ?? 100000 rows, each carrying multi-KB vector BLOBs, into JS memory in one synchronous call (hence 4.2 GB RSS and the JS_all stack above).
• Decode + cosine also run synchronously on the main thread.

With a table in the hundreds-of-thousands of rows, every dreaming/retrieval cycle blocks the event loop long enough to peg a core and make the gateway unresponsive.

Suggested fixes: add a real vector index (e.g. sqlite-vec), or at minimum (a) stream/iterate instead of .all(), (b) do decode+cosine off the main thread (worker), (c) pre-filter candidates by cheap SQL predicates before loading BLOBs, (d) make the default hardCap much smaller and explicit.

Bug #2 — dedup pagination cap → unbounded re-insertion

Documented in a code comment at dist/core/storage/repos/traces.js (listAllForEpisode):

/**
 * Full episode-scoped trace fetch with NO pagination cap. The paginated
 * `list({ episodeId })` path silently truncates to 500 rows (default 50),
 * which breaks capture-side dedup when an episode has more than the cap
 * worth of steps — the next runLite/runReflect re-inserts everything past
 * the cap as "novel". Use this for any dedup / reconciliation read.
 */

If any code path still feeds dedup from the paginated list({ episodeId }) (cap 500) instead of listAllForEpisode, then for any episode with >500 steps the dedup check cannot see the older traces, so runLite/runReflect treats them as novel and re-inserts them every cycle.

Observed in our DB — the damage is exactly this shape:

• traces: 518,375 rows, but only 80,583 distinct by (episode_id, turn_id, user_text, agent_text, tool_calls_json) → 437,792 exact duplicates (84%).
• Duplicates concentrated in a few large episodes, e.g. one episode had 49,402 rows but only 153 distinct turn_ids / 588 distinct contents — i.e. the same turns re-inserted hundreds of times.
• Duplicates span the full lifetime of the DB (2026-05-19 → 2026-07-08), consistent with a per-cycle re-insertion loop rather than a one-off.

Suggested fix: ensure all capture-side dedup/reconciliation reads use the uncapped listAllForEpisode (or a keyset-paginated full read), and/or add a UNIQUE constraint / INSERT … ON CONFLICT DO NOTHING on the natural trace identity so re-insertion is idempotent.

Impact

• Gateway CPU pinned at 100%, unresponsive, 4.2 GB RSS.
• DB bloated 8× (≈850 MB → 6.8 GB) purely from duplicate rows + their vectors.

Workaround we applied (data-side only — does not fix the plugin bugs)

Stopped the gateway, backed up, de-duplicated traces by full content identity (518,375 → 80,583 rows), rebuilt the traces_fts FTS5 index, VACUUMed (6.8 GB → 1.3 GB), restarted. CPU settled to ~11%. But both root causes are in plugin code, so this will recur as data grows again.

One operational note that bit us during cleanup: traces has trigram FTS5 triggers (traces_fts_ai/ad/au), so a row-by-row DELETE of 437k rows is pathologically slow (didn't finish in 50 min). Dropping the triggers, doing the bulk delete, then rebuilding the FTS content and recreating the triggers was orders of magnitude faster. Not a bug per se, but worth being aware of for any maintenance tooling.

Happy to provide more detail or test a fix.