Compaction silently discards all tool outputs and assistant reasoning

After reading through compact.rs, I ran a reproducible experiment to quantify the information loss. The results are significant enough to warrant a structural discussion.

How compaction works (code evidence)

collect_user_messages() is the only filter applied to history before rebuilding the compacted context:

pub(crate) fn collect_user_messages(items: &[ResponseItem]) -> Vec<String> {
    items.iter().filter_map(|item| match parse_turn_item(item) {
        Some(TurnItem::UserMessage(user)) => Some(user.message()),
        _ => None,  // ALL other item types dropped here
    }).collect()
}

build_compacted_history_with_limit() then reconstructs history from:

≤20,000 tokens of user messages (recency-weighted, truncated)
One LLM-generated summary (last assistant message from compaction turn)
Ghost snapshots (preserved separately)

Every other item type is permanently dropped: assistant messages, tool calls, tool results, file contents, bash outputs, patch diffs, test results.

Experiment: information retention across compactions

I simulated a realistic 6-turn coding session (auth bug fix: read files → run tests → apply patch → regression test) and ran it through the same logic as compact.rs. The script is fully reproducible — it mirrors collect_user_messages() and build_compacted_history_with_limit() exactly.

Session composition before any compaction (25 items, 1,670 tokens):

Item type	Count	Tokens	Share
`tool_result` (file reads, bash output, patches)	7	1,324	79.3%
`assistant_message` (reasoning, analysis, plans)	6	221	13.2%
`user_message`	5	71	4.3%
`tool_call`	7	54	3.2%

After 1st compaction (6 items, 229 tokens — 13.7% retention):

Item type	Count	Tokens	Share
`user_message`	5	71	31.0%
LLM summary	1	158	69.0%
`tool_result`	0	0	0%
`tool_call`	0	0	0%

After 2nd compaction (115 tokens — 6.9% of original):

All 5 critical facts checked for presence in surviving context:

Fact	After 1st	After 2nd
`cache::invalidate` fix (the actual bug fix)	✅	❌
`test_concurrent_refresh` failure (confirmed root cause)	✅	❌
`user_42` regression test content	❌	❌
`mobile.rs` also calls the buggy path	❌	❌
Line 47 in auth.rs (location of the fix)	❌	❌

Key findings

Tool results are 79% of session tokens but 0% survive compaction. These are the highest-density items in a coding session — file contents, test output, applied diffs.
"Telephone game" degradation: 1st compaction retains 13.7%. 2nd compaction of that retains 50% → 6.9% of the original. Each compaction summarizes the previous summary.
The system acknowledges this is a problem — the post-compaction warning reads: "Long threads and multiple compactions can cause the model to be less accurate." But the warning fires after the information is already gone.
No cross-session persistence: when the session ends, everything is lost regardless of compaction. The next session starts blank.

Root cause: compaction is purely in-context compression

The design treats memory as a context-window management problem: compress when full, discard the rest. This works for short, self-contained tasks. It breaks for:

Long coding sessions (hours, multiple compactions)
Tasks where exact intermediate state matters (what file was at line 47, which tests passed)
Multi-session work on the same codebase

Proposed direction: Memory MCP Server

Rather than compressing everything into a lossy in-context summary, give the model explicit tools to write important facts to durable external storage and read them back selectively:

memory_save(key, value, tags)   — persist a fact, decision, or file snapshot
memory_recall(query)            — semantic search over saved memories
memory_checkpoint(label)        — snapshot current task state  
memory_list(tag_filter)         — enumerate saved keys

This decouples memory from the context window:

The model actively chooses what to persist (not passive compression)
Memories survive session end
Only relevant context is retrieved on demand (not a full dump)
Users can inspect and edit what the model remembers

A local-first implementation (SQLite + embeddings) could ship with zero cloud dependencies and integrate naturally with the MCP infrastructure Codex already supports. Compaction would still exist for context management, but critical facts would be durably saved before the window fills.

Happy to prototype this if the direction is of interest. Opening here to gauge appetite before writing code.

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

Powered by Codex Action

Follow-up after reading the existing memories/ implementation

After digging into codex-rs/core/src/memories/, I want to be more precise about the gap — because the existing system is actually quite good and covers a lot.

What the existing memory system already handles

The Phase 1/2 pipeline is well-designed for cross-session learning:

Phase 1 (at startup): scans recent rollouts, extracts durable lessons per thread using a structured prompt, stores in state DB
Phase 2: consolidates into memory_summary.md, injected into subsequent session system prompts
MemoryTool (when enabled): model can actively read MEMORY.md + rollout summaries during a turn, and is instructed to write updates back when it detects stale guidance

This handles the "learn across sessions" problem well.

The specific gap: compaction timing

The remaining problem is narrower: the model cannot proactively write to memory before compaction replaces its history.

The flow is:

context fills → compact.rs triggers automatically → history replaced → MemoryTool could help now, but facts are already gone

Phase 1 explicitly excludes exactly the facts that are lost:

"Temporary facts (live metrics, ephemeral outputs) that should be re-queried"
"Generic status updates ('ran eval', 'looked at logs') without takeaways"

But these "temporary facts" are precisely what a multi-hour coding session needs to remain coherent within that session:

which test failed at step 3 and why
what the file actually contained at line 47 when the decision was made
the exact patch that was applied and confirmed working
which API path was already ruled out

These aren't durable cross-session lessons — they're session-local technical state that the MemoryTool is also not designed for (it targets long-term reusable knowledge).

Proposed: pre-compaction session snapshot

One targeted fix: before replace_compacted_history() discards history, run a lightweight pass that writes session-local technical state to a structured, temporary location (e.g., ~/.codex/sessions/<id>/checkpoint.md).

Unlike Phase 1 (which extracts lessons), this checkpoint would:

preserve verbatim the last N tool results (file contents, test output, diffs)
not summarize — exact content, not impressions
auto-expire at session end (not fed into Phase 1)
be injected back after compaction via the same developer instructions mechanism already used for memory_summary.md

The model would see its own technical context restored after compaction instead of discovering mid-task that it no longer knows what files it changed.

This complements rather than replaces the existing system: Phase 1/2 handles cross-session learning, the session snapshot handles intra-session continuity.

I can prototype this as an extension to the existing run_compact_task_inner() flow if there's appetite for it.

Prototype + evaluation results for the pre-compaction snapshot approach

Built a prototype of extract_session_snapshot() and ran L1/L2 metrics against the same coding session from the original report. Sharing the numbers before writing any Rust.

How the snapshot works

Before replace_compacted_history() discards history, scan tool results and classify them:

Signal	Fact type	Importance
Diff hunks (`---`/`+++`/`@@`)	`PATCH`	1.0
Test output (`FAILED`/`ok`/`panicked`)	`TEST_RESULT`	0.95
Source file content (`pub fn`, `impl`, etc.)	`FILE_CONTENT`	0.80
Assistant messages with decisions/findings	`DECISION`	0.70
Other command output	`COMMAND_OUTPUT`	0.60

Items ranked by importance and packed into a fixed token budget (e.g. 2,000 tokens), then appended to the post-compaction context via the same developer instructions mechanism already used for memory_summary.md.

L1 — Weighted Fact Recall

8 ground-truth facts extracted from the session (file paths, line numbers, test results, patch content, decisions):

Fact	Baseline	+ Snapshot
fix location: `src/auth.rs` line 47	✅	✅
root cause: missing `cache::invalidate()`	❌	✅
`test_concurrent_refresh`: failed → passed	❌	✅
regression test content (`user_42`)	❌	✅
`mobile.rs:89` also calls `refresh_token`	❌	✅
exact patch: `+cache::invalidate(&decoded.sub)`	❌	✅
test count: 14 total, 13+1 originally	❌	✅
`web.rs:67` also in scope	❌	✅

Weighted recall: 15.2% → 100.0% (+84.8%)

L2 — Context Efficiency

Snapshot uses a fixed 2,000-token budget regardless of session length
Compression ratio vs original tool results: ~1.5x (snapshot is slightly larger than raw because of structure/labels — tradeoff for readability)
Context overhead scales inversely with session size: at a realistic 10K-token post-compaction context the overhead is ~20%; at 20K it drops to ~10%

L3 — Task continuation quality (planned)

The static metrics above don't capture whether a model actually uses the checkpoint correctly. The next step is a live evaluation:

4 questions asked to the model after compaction (with vs without snapshot)
Scored on a 0–3 rubric (0=hallucinated, 1=vague, 2=partial, 3=exact)
Hypothesis: baseline avg ~0.8–1.2, snapshot avg ~2.4–2.8

Will run this once there's signal that the direction is worth pursuing — happy to share results here.

No code changes proposed yet — wanted to validate the idea empirically before touching compact.rs. Let me know if this is a direction that fits with the roadmap.

Reviewed all five — not duplicates, but #14347 is the closest related issue and worth linking.

#14347 ("Extend compaction prompt") addresses the same symptom (information loss across compactions) but from a different angle: improve the LLM summarization prompt so it captures more. That is a valid fix for reasoning and decisions.

The gap this issue focuses on is complementary and lower in the stack: even a perfect compaction prompt cannot recover tool outputs, because collect_user_messages() structurally discards them before the LLM ever sees them for summarization:

pub(crate) fn collect_user_messages(items: &[ResponseItem]) -> Vec<String> {
    items.iter().filter_map(|item| match parse_turn_item(item) {
        Some(TurnItem::UserMessage(user)) => Some(user.message()),
        _ => None,  // tool_results, assistant_messages: dropped here, not summarized
    }).collect()
}

The compaction model is only given user text as input. File contents, test output, applied patches — they are not passed to the summarizer at all. A better prompt cannot help with data it does not receive.

So the two issues address different root causes:

Extend compaction prompt to reduce loss over multiple compactions #14347 → improve what the LLM extracts from what it does receive (user messages + previous summary)
This issue → preserve what the LLM never receives (tool results, assistant reasoning)

The pre-compaction snapshot proposal would complement #14347's prompt improvements rather than replace them.

#13279 and #14120 are about the compaction retry loop, which #14563 just fixed — different problem.
#13932 is about plan preservation specifically — a subset of the broader tool output loss described here.
#13142 is a feature request for user-configurable compaction routines — related direction, different mechanism.

Add session-memory skill: preserve technical facts across compaction

Codex is working as intended here, so we don't consider this a bug.

Navigation Menu

Repository navigation

Compaction silently discards all tool outputs and assistant reasoning #14589