Dreamer map-memories / curate: validation contract defects make format variance unrecoverable (and can silently corrupt mappings) #323

Package: @cortexkit/pi-magic-context 0.34.2 (Pi plugin build)
Also verified against: upstream master @ 652a42ed7ecd9e2438ba31ab30fcbeffe07c8191 (2026-08-16) — MAP_MEMORIES_SYSTEM_PROMPT is byte-identical (SHA-256 ef1c6a008af07e897f044c1ff6c0537c2954fa2cd8bbe16d4b801a782276feae), and validateOutput still calls only parseMapMemoriesManifest(text) while assertManifestCoversExactly() remains in applyBatchMappings(). v0.37.0 release notes do not mention a fix.

Summary

Running the production map-memories prompt with four different models (Gemini 3.7 Flash High, DeepSeek V4 Flash, GLM-5.2, GLM-5.3 — all with reasoning enabled, read-only tools working, routing confirmed) produced four different manifest deviations, and the host validation chain mishandled every one of them:

• wrong-but-rooted manifests (<map>, JSON array, <mapping>) pass validation as an empty parse, so the designed fallback-model recovery never fires;
• the exact id-coverage check runs after the retry window has closed;
• one deviation shape (nested <file> children) is silently persisted as independent=true — a memory with real backing files gets downgraded with no error at all;
• curate accepts a textual pseudo-tool-call as task completion, because its only validation is "latest assistant text is non-empty".

The same codebase already contains the correct pattern in the Historian pipeline (validateHistorianOutput requires a non-empty parse + semantic coverage checks, and buildHistorianRepairPrompt feeds the validation error back to the same model before falling back). The Dreamer modules simply don't use it.

We believe this is primarily a host-side contract problem, not a model-capability problem: the same four models pass the Historian's format contract (which is leniently parsed and repaired on failure) while all four fail the map-memories contract (which is under-specified in the prompt and then validated at the wrong layer).

Defect 1 — parseMapMemoriesManifest() accepts an empty manifest; validateOutput therefore never triggers fallback

Where: packages/plugin/src/features/magic-context/dreamer/map-memories.ts — parseMapMemoriesManifest() (0.34.2 dist bundle dist/index.js:7606-7624), called from validateOutput at dist/index.js:7718-7727.

What happens: after extractCompleteManifestBody(text, "mappings") finds a well-formed <mappings>…</mappings> root, the parser loops over <memory …> matches. If the model used any other element name, the loop matches zero times, no error is thrown, and an empty array is returned. validateOutput treats this as success, so promptSyncWithValidatedOutputRetry() marks the primary model "validated" and never consults fallback_models.

Minimal repro (real model outputs from an isolated canary, production prompt byte-identical):

Gemini 3.7 Flash High:

<mappings>

<map id="1">

config/magic-default.jsonc

</map>

</mappings>

GLM-5.2:

<mappings>

[

{ "id": "CONFIG_VALUES", "files": ["config/magic-default.jsonc"] }

]

</mappings>

GLM-5.3:

<mappings>

<mapping id="CONFIG_VALUES">

<files><file path="config/magic-default.jsonc">…</file></files>

<status>confirmed</status>

</mapping>

</mappings>

All three parse to [] → validation passes → the batch later fails in applyBatchMappings() with the retry chain already closed (mapOneBatch catch records failed and returns {mapped: 0, independent: 0}).

Suggested fix: in validateOutput, require a non-empty parse and run the exact id-coverage assertion there (see Defect 2), so any of these shapes throws inside the retry boundary.

Defect 2 — assertManifestCoversExactly() runs outside the retry window

Where: packages/plugin/src/features/magic-context/dreamer/map-memories.ts — applyBatchMappings() (dist index.js:7751-7754); the assertion itself is in packages/plugin/src/features/magic-context/dreamer/manifest-parser.ts (dist index.js:4379-4389).

What happens: the only place the manifest is checked for exact id coverage is after promptSyncWithValidatedOutputRetry() has already returned success. A coverage failure can therefore never be retried — neither with the same model nor with fallbacks — even though validation-fallback is the module's designed recovery mechanism (see PR #175 lineage).

Suggested fix: move the coverage assertion into validateOutput (it has access to the batch via closure), or pass expected ids into the parser with a requireCoverage option.

Defect 3 — missing files attribute silently becomes independent=true (data corruption, no error)

Where: parseMapMemoriesManifest(), dist index.js:7620:

out.push({ id, files, independent: independent || files.length === 0 });

What happens: DeepSeek V4 Flash produced:

<mappings>

<memory id="1" name="CONFIG_VALUES">

<file path="config/magic-default.jsonc" claim="…" verified="true" line="14"/>

</memory>

</mappings>

The opening tag matches the regex, id="1" is captured, id coverage passes — but the top-level <memory> has no files attribute, so files.length === 0 forces independent: true. A memory that the model explicitly backed with a real file is persisted as file-independent, silently. This is worse than an explicit failure: the store is now wrong and nothing surfaces it.

Suggested fix: only honor an explicit independent="true" attribute; treat <memory> elements with children, or with neither files nor independent, as a hard validation error. Optionally accept a nested <file path="…"/> form as a documented alias and normalize it, rather than discarding it.

Defect 4 — no error-feedback repair pass (the Historian pipeline already has one)

Where: packages/plugin/src/shared/model-suggestion-retry.ts — promptSyncWithValidatedOutputRetry() (dist index.js:3985-4049). With fallback_models: [], a validation failure throws immediately (dist index.js:4008-4010); the same model is never re-asked with the error.

Contrast that proves the pattern exists in this codebase: packages/plugin/src/hooks/magic-context/compartment-runner-validation.ts — validateHistorianOutput() (dist index.js:18605-18636) fails closed on an empty parse and performs semantic coverage checks; on failure, buildHistorianRepairPrompt() (dist index.js:18658-18673) sends the validation error plus the previous invalid output back to the same model (repair pass at dist index.js:20972), and only then are fallback models tried (dist index.js:21001). The Historian also parses leniently — pure regex over the whole text, no single-root requirement (packages/plugin/src/hooks/magic-context/compartment-parser.ts, parseCompartmentOutput), which is why code fences and wrapper elements don't break it.

Suggested fix: give promptSyncWithValidatedOutputRetry() (or the map-memories caller) an optional same-model repair round that receives the validation error message and the previous output — mirroring the Historian flow — before consuming fallback models.

Defect 5 — prompt/tool-surface mismatch, and an under-specified output contract

Where:

• packages/plugin/src/features/magic-context/dreamer/map-memories-prompt.ts (dist index.js:7543-7565) tells the model: Tools (read-only): read, grep, glob, aft_search, aft_outline, aft_zoom.
• The strict tool allowlist actually granted to dreamer-memory-mapper is read, grep, find, ls, aft_outline, aft_zoom, aft_search — no glob, plus undocumented find/ls (packages/pi-plugin/src/subagent-runner.ts, PI_READ_ONLY_BUILTINS + STRICT_TOOL_ALLOWLIST, dist index.js:2275-2276, 2301).

What happens: the prompt advertises a tool that does not exist in the child session. Beyond the practical confusion, a model that notices the tool list is inaccurate has evidence that the prompt is approximate — and treats the output grammar as approximate too. Compounding this, the entire output specification is a 4-line skeleton with placeholders (<memory id="N" files="path/a.ts,path/b.ts"/>) and no realistic few-shot example; compare the Historian system prompt, which embeds dozens of full exemplar compartments. Four different models inventing four different schemas is the expected outcome of an under-specified contract, not four coincidental model failures. (Note: the prompt comment says it was calibrated with DeepSeek-v4-Flash in a shadow harness — yet DeepSeek V4 Flash itself deviated, so the calibration does not generalize.)

Suggested fix: align the tool list with the actual allowlist (or vice versa); add 1–2 complete few-shot manifests with real-looking ids and paths; consider explicitly listing the only two permitted element shapes.

Defect 6 — curate / maintain-docs completion check accepts pseudo-tool-call text

Where: packages/plugin/src/features/magic-context/dreamer/task-executor.ts — runAgenticTask() validateOutput (dist index.js:10096-10100):

validateOutput: (messages) => {

const text = extractLatestAssistantText(messages);

if (!text)

throw new Error("Dreamer returned no assistant output.");

return text;

}

What happens: in a canary with the production CURATE_SYSTEM_PROMPT and a mocked ctx_memory tool, Gemini 3.7 Flash High correctly executed four real tool calls (merge / update / archive / archive), then emitted its fifth action as plain text:

归档与全局用户画像完全重复且无项目特化信息的记忆条目。[historical tool call]

id: call_2080315

name: ctx_memory

arguments:

{"action":"archive","reason":"…","ids":[6]}

The run was recorded completed; memory 6 stayed active and two further memories were never processed. Scheduled maintenance must not silently skip work.

Suggested fix: validate that the latest assistant message contains no unresolved tool intent (e.g. reject text matching pseudo-call patterns when the task expects tool actions), and/or reconcile the pool state after the run (every in-scope memory must have been either acted on via a real tool call or explicitly left with a stated reason) before recording completed.

Suggested split

This can be landed as one umbrella fix or split into: (A) map-memories validation chain (Defects 1–4), (B) prompt/tool-list alignment (Defect 5), (C) agentic-task completion validation (Defect 6). Defect 3 is the only one that silently corrupts persisted state and deserves priority.

Notes on evidence

• All model outputs above come from an isolated same-sample harness using the byte-identical production system prompt (SHA-256 verified), synthetic memories, and read-only tools; no real session content, databases, or project configs were touched.
• Historian control group: with the same four models plus two others, the Historian/Editor two-pass pipeline parsed successfully in 8/8 calls with zero factual omissions — evidence that the models can meet a well-specified, leniently-validated, repairable contract.
• Formatting note: angle brackets in evidence samples are HTML-entity-escaped because the submission pipeline strips raw markup; the rendered issue shows them correctly.