v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB

Problem

All tagged plugin versions from v2.0.2 through v2.0.5 hang indefinitely during bootstrapMemoryCoreFull() when the SQLite database exceeds ~500MB. The bridge process burns 80-157% CPU (userspace, WCHAN=0) and never reaches migration.applied or pipeline.ready.

Reproduction

Run any bridge from v2.0.2, v2.0.4, or v2.0.5 against a memos.db >500MB
Observe: sqlite.open appears in logs, then silence
CPU stays at 80-157% with no further log output
Bridge never reaches migration.applied

Bisect Results

Version	Result	Notes
v2.0.5	❌ Hang	687MB DB, 80-157% CPU
v2.0.4	❌ Hang	Same symptom
v2.0.2	❌ Hang	Same symptom
v2.0.0-beta.5	⚠️ Boots	Boots fast but heavy orphan processing
v2.0.0-beta.1	✅ Clean	18ms boot, all subsystems initialize

Regression window: Between v2.0.0-beta.5 and v2.0.2.

Environment

DB: 687MB, WAL mode, ~3,500 traces
The host gateway kills and respawns the bridge every ~30s because it never becomes healthy
Smaller databases (tested: 43MB, 56MB) do NOT trigger the hang

What This Is NOT

NOT the two-bridge SQLite lock race (that produces database is locked errors; this produces silence + CPU)
NOT the abandoned-episode scoring gap (Reward pipeline skips abandoned episodes — 98% of closed episodes never scored #1782) — those fixes address the scoring pipeline, not bootstrap

Suspected Cause

bootstrapMemoryCoreFull() runs synchronously during bridge init. Something inside it scales poorly with database size — candidate: full table scans during migration checks, unbuffered trace iteration, embedding pipeline loading all traces, or orphan detection walking the entire table.

Current Workaround

Roll back to v2.0.0-beta.1. It boots cleanly on the same 687MB database.

Root Cause Confirmed

Investigation via bisect and SQLite inspection confirms:

Migration 007 (namespace-visibility) is the root cause. It runs these operations on the traces table inside a single db.tx():

UPDATE traces SET share_scope='private' WHERE share_scope IS NULL;
CREATE INDEX IF NOT EXISTS idx_traces_owner ON traces(owner_agent_kind, owner_profile_id, ts DESC);
CREATE INDEX IF NOT EXISTS idx_traces_share ON traces(share_scope, ts DESC);

On a 687MB database with ~98,000 traces, the UPDATE (rewriting every row that has share_scope IS NULL — which is every row from pre-007 schemas) plus two full-table index builds exceed the host gateway's 30-second kill timeout. SQLite rolls back the entire transaction, including the schema_migrations INSERT, so migration 007 is never recorded and retries on every boot — infinite crash-loop.

Small databases (~43MB) complete within the timeout, which is why this only manifests on larger installs.

Fix Approach

Two-phase migration:

Phase 1 (inside transaction, milliseconds): ALTER TABLE ... ADD COLUMN only — metadata operations regardless of DB size. The schema_migrations record for v7 commits here.
Phase 2 (after migration loop, outside any transaction): Batched UPDATE WHERE share_scope IS NULL AND rowid IN (SELECT rowid ... LIMIT 2000) in 2,000-row chunks (each its own implicit transaction) + CREATE INDEX IF NOT EXISTS (idempotent).

Restart safety: If the bridge is killed during Phase 2, the v7 schema_migrations record survives. Next boot skips Phase 1 and resumes Phase 2 where it left off. The crash-loop is broken.

Precedent: The codebase's own ensureHubSharingSearchColumns() already runs outside the migration loop in runMigrations() — this extends that pattern to the namespace migration.

Also Affected

Migration 012 (v2.0.5) adds two more CREATE INDEX on the traces table inside a db.tx(). On a large DB, these may also exceed the kill timeout. The same two-phase treatment would apply.

Verification

DB confirmed at migrations 1-6 only (007 never committed)
No corruption — all failed transactions rolled back cleanly
The fix has been tested locally and the bridge boots successfully

⏸️ AutoDev task 4564693dff511257 needs manual clarification.

Reason: 已修复 #1787 引导挂起回归并 push 到 autodev/MemOS-1787。请确认是否创建 PR。

PR #1789 already implements this, autodev branch can be discarded.

🤖 AutoDev has picked up this issue and started working on it.

Task ID: 4c6bfd1856a05a4c
Working branch: autodev/MemOS-1787
Target branch: main
Workflow: opsp (analysis → coding → testing → PR)

I will post the PR link here once done. If I need more information, I will ask in the comments.

fix(memos-local-plugin): avoid bootstrap hang on large databases (#1787)

🚀 AutoDev task 4c6bfd1856a05a4c pushed commit.

Summary: Successfully fixed the v2.0.2+ regression where bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB.

Root Cause:
The namespace-visibility migration was issuing a bulk UPDATE on all owner-aware tables, including the traces table which is the largest table in busy installations. On databases past ~500 MB, this UPDATE held the synchronous bootstrap transaction in CPU-bound row rewriting (re-validating JSON CHECK constraints on every row) for many minutes and never reached migrations.summary. Additionally, the startup dirty-closed-episode scan was calling getManyByIds(traceIds).some(tr => tr.ts > scoredAt), which hydrated every column of every trace (embedding BLOBs, full tool_calls_json, agent text) into Node memory just to inspect a single timestamp.

Fix Applied:

Removed the bulk UPDATE from the migration in migrator.ts (line 266) - the application layer already treats NULL share_scope as 'private' via normalizeShareScope and COALESCE in visibilityWhere, and new rows get the column DEFAULT, so the bulk UPDATE was purely cosmetic.
Added traces.hasAnyNewerThan(ids, ts) helper in repos/traces.ts that issues a single SELECT 1 ... LIMIT 1 per chunk instead of hydrating full trace rows.
Updated memory-core.ts to use the new lightweight helper instead of getManyByIds().some().

Tests Added:

Regression test in migrator.test.ts that verifies NULL share_scope rows stay NULL after migration (would flip to 'private' if the bulk UPDATE still existed)
Coverage for traces.hasAnyNewerThan in repos.test.ts with boundary condition testing

Files Changed:

apps/memos-local-plugin/core/storage/migrator.ts
apps/memos-local-plugin/core/storage/repos/traces.ts
apps/memos-local-plugin/core/pipeline/memory-core.ts
apps/memos-local-plugin/tests/unit/storage/migrator.test.ts
apps/memos-local-plugin/tests/unit/storage/repos.test.ts

The fix eliminates the O(n) row rewrite on large trace tables during bootstrap and replaces the O(total trace bytes) scan with an O(chunk size) exists-check, resolving the hang reported in #1787.

Base branch: main
Branch: autodev/MemOS-1787
Commit: b2b474b

I will create the PR next and post the PR link here.

@MatthewZhuang

✅ AutoDev task 4c6bfd1856a05a4c completed.

Summary: Successfully fixed the v2.0.2+ regression where bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB.

Root Cause:
The namespace-visibility migration was issuing a bulk UPDATE on all owner-aware tables, including the traces table which is the largest table in busy installations. On databases past ~500 MB, this UPDATE held the synchronous bootstrap transaction in CPU-bound row rewriting (re-validating JSON CHECK constraints on every row) for many minutes and never reached migrations.summary. Additionally, the startup dirty-closed-episode scan was calling getManyByIds(traceIds).some(tr => tr.ts > scoredAt), which hydrated every column of every trace (embedding BLOBs, full tool_calls_json, agent text) into Node memory just to inspect a single timestamp.

Fix Applied:

Removed the bulk UPDATE from the migration in migrator.ts (line 266) - the application layer already treats NULL share_scope as 'private' via normalizeShareScope and COALESCE in visibilityWhere, and new rows get the column DEFAULT, so the bulk UPDATE was purely cosmetic.
Added traces.hasAnyNewerThan(ids, ts) helper in repos/traces.ts that issues a single SELECT 1 ... LIMIT 1 per chunk instead of hydrating full trace rows.
Updated memory-core.ts to use the new lightweight helper instead of getManyByIds().some().

Tests Added:

Regression test in migrator.test.ts that verifies NULL share_scope rows stay NULL after migration (would flip to 'private' if the bulk UPDATE still existed)
Coverage for traces.hasAnyNewerThan in repos.test.ts with boundary condition testing

Files Changed:

apps/memos-local-plugin/core/storage/migrator.ts
apps/memos-local-plugin/core/storage/repos/traces.ts
apps/memos-local-plugin/core/pipeline/memory-core.ts
apps/memos-local-plugin/tests/unit/storage/migrator.test.ts
apps/memos-local-plugin/tests/unit/storage/repos.test.ts

The fix eliminates the O(n) row rewrite on large trace tables during bootstrap and replaces the O(total trace bytes) scan with an O(chunk size) exists-check, resolving the hang reported in #1787.

Base branch: main
Branch: autodev/MemOS-1787
Commit: b2b474b
PR: #1871
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219

Fix #1787: v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 1…

fix(memos-local-plugin): tolerate ghost trace IDs in reward dirty che…

Fix #1966: bug: ghost trace in episode.trace_ids_json causes infinite…

Fix MemTensor#1787: v2.0.2+ regression: bootstrapMemoryCoreFull() han…

Fix MemTensor#1966: bug: ghost trace in episode.trace_ids_json causes…

Navigation Menu

Repository navigation

v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB #1787

Problem

Reproduction

Bisect Results

Environment

What This Is NOT

Suspected Cause

Current Workaround

Related

Root Cause Confirmed

Fix Approach

Also Affected

Verification

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB #1787

Description

Problem

Reproduction

Bisect Results

Environment

What This Is NOT

Suspected Cause

Current Workaround

Related

Activity

chiefmojo commented on May 23, 2026

Root Cause Confirmed

Fix Approach

Also Affected

Verification

Memtensor-AI commented on May 26, 2026

chiefmojo commented on May 26, 2026

Memtensor-AI commented on Jun 3, 2026

Memtensor-AI commented on Jun 3, 2026

Memtensor-AI commented on Jun 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions