Repository navigation
Context remaining/token estimate overcounts MCP image data URLs (base64) #11845
Description
Summary
When an MCP tool returns an image, the client converts it to ContentItems containing InputImage { image_url: "data:...;base64," } and stores that in history. Token/context estimation uses the serialized JSON length of ResponseItem (including that base64 payload), so a large image makes the estimated tokens jump and context remaining drops sharply, sometimes triggering auto-compaction.
Code pointers
- codex-rs/protocol/src/models.rs: convert_mcp_content_to_items builds data URLs for MCP image content (type="image").
- codex-rs/core/src/context_manager/history.rs: estimate_response_item_model_visible_bytes falls back to serde_json::to_string(item).len() for most items, then approx_tokens_from_byte_count_i64.
- codex-rs/core/src/codex.rs: Session::recompute_token_usage uses estimate_token_count_with_base_instructions to set last_token_usage.total_tokens.
Why this is wrong
The model consumes the image as an image, not as the base64 text inside the data URL. Counting base64 bytes as model-visible text massively overestimates context usage.
Repro
- Configure any MCP server tool that returns an image result with a large base64 payload (type="image").
- Call the tool from Codex and let it return the image.
- Observe context remaining percentage and estimated tokens jump disproportionately right after the tool output.
Expected
Image size should not dominate token estimation. Estimation should treat InputImage as fixed-cost or capped-cost, or strip the base64 payload when estimating.
Environment
- openai/codex @ ebceb71
Activity
miraclebakelaser commentedon Feb 15, 2026
I'm guessing this explains #11072
lucamorettibuilds commentedon Feb 18, 2026
Opened #12104 with a fix for this. The core issue is in
estimate_response_item_model_visible_bytes()— it counts the full serialized JSON length ofInputImageitems, including the raw base64 payload. A single 100KB screenshot gets estimated at ~25K tokens when the model actually processes it as a fixed-cost vision tile (~85 tokens).The fix detects
data:image/...;base64,URLs insideContentItem::InputImageandFunctionCallOutputContentItem::InputImage, subtracts the base64 payload size, and adds a fixed estimate (340 bytes ≈ 85 tokens) matching typical vision model tile costs. Added tests for both the message and function call output paths.fix: exclude base64 image data URLs from context token estimation
Improve token usage estimate for images (openai#12419)
miraclebakelaser commentedon Feb 24, 2026
@etraut-openai thank you for fixing this.
One small issue: 340 bytes (~85 tokens) is too low of an estimate.
codex/codex-rs/core/src/context_manager/history.rs
Lines 422 to 426 in 0679e70
The
gpt-5.2image cost is determined by this formula[1][2]:image_tokens = ceil(1.2 * min( ceil(resized_width/32) * ceil(resized_height/32), 1536 ))In other words,
gpt-5.2image inputs are capped at 1844 tokens (1536 * 1.2), which is ~7373 bytes. Thegpt-5.3-codeximage cost calculation is not published, so I'm assuming it's in the ballpark. Would be glad to be corrected by someone on your team.Given that these estimates affect
total_usage_tokens, which is used in auto-compaction logic, we should use 1844 tokens as our estimate. The downside of using too low of an estimate is that auto-compaction could fail due to insufficient free context in conversations that contain many images.Weighing the pros and cons of this change, the con is that auto-compaction may trigger marginally earlier. The pro is that you avoid session-poisoning compaction failures.
My proposed change is:
Can I make a PR for this?
Raise image byte estimate for compaction token accounting (#12717)
Raise image byte estimate for compaction token accounting (#12717)
Improve token usage estimate for images (openai#12419)
Raise image byte estimate for compaction token accounting (openai#12717)
Raise image byte estimate for compaction token accounting (#12717)