Repository navigation
Let the model replace a junk tool result with a one-line summary (model-judged context pruning) #4715
Description
What feature would you like to see?
TL;DR: When a tool call returns junk the model didn't intend, let the model swap that output for a one-line summary before the turn ends.
Pay for it once; keep the context clean for the rest of the session.
Important
Why prevention can't fix this
You open a file or run a command because you don't know what's in it, so its output size can't be predicted in advance.
A human glances at junk and looks away. A model re-reads it on every turn.
This gives the model the ability to look away. 🙈
Sequence
TODAY
turn N grep -r ... ──► ~40 KB of target/ build fingerprints
model ──► "oops, irrelevant" (but can't drop it)
turn N+1..50 every request ──► re-sends the same ~40 KB ❌ cost + diluted attention
PROPOSED
turn N grep -r ... ──► ~40 KB
model ──► prune_tool_result(call_id, "grep hit target/ fingerprints, irrelevant")
context now ──► [pruned] grep hit target/ fingerprints, irrelevant (1 line)
turn N+1..50 every request ──► 1 line ✅
Why it's cheap: only the tail changes
[ instructions | turns 1 .. N-1 | junk output | rest of turn N ]
└───── cached prefix: kept ──────┘ └── re-cached once (small) ──┘
Prompt caching is prefix-based, so pruning a result from the current turn leaves everything before it cached.
Design
| Rule | Why |
|---|---|
| Model-judged | "This is junk" is a semantic call; no size cap or rule can make it. |
| Tail-only (current turn) | The cached prefix survives; nothing old is rewritten. |
| Visible | Original stays in the user-facing transcript with a one-line notice. Nothing disappears silently. |
Bonus: a bigger effective window
Removing junk grows the useful space by more than the junk's share, because the denominator shrinks:
| Junk share of context | Useful space | Effective window gain |
|---|---|---|
| 10% | 90% → 100% | +11% |
| 20% | 80% → 100% | +25% |
| 30% | 70% → 100% | +43% |
| 50% | 50% → 100% | +100% |
Same hardware, same model: compaction comes later, and what remains is signal. (Junk shares are illustrative.)
Additional information
| Mechanism | Gap |
|---|---|
| Micro-compaction (blanks old tool results) | Rule-based and age-based: the newest result, exactly where accidental junk sits, is kept. |
| Compaction / clear / undo | Periodic or manual and coarse; junk rides along until then. |
| Output truncation | Cut by size, not relevance. |
"This output is junk" is a semantic judgment only the model can make, at the moment the output arrives.
Everyone wins 🌱
| Who | What they get |
|---|---|
| Users | Fewer tokens burned per task |
| The model | A clean context, better answers |
| Providers | Less KV cache per session, more sessions per GPU |
| The planet | Less power, fewer GPUs to build |
Junk is pure overhead: no answer improves because a build-fingerprint dump was re-read many times. Drop it in every harness, and the whole industry does the same work with less. The cheapest watt is the one you never spend.
Filed across 17 agent harnesses; full list: https://gist.github.com/nicezic/69f13fde6fa080a571825b56fd76392d
Activity