Repository navigation
Codex should not resend large base64 image tool outputs in subsequent context #28316
Description
Summary
When a user submits an image to Codex, sending the image in the immediate request is expected. However, after that turn is processed, Codex appears to persist the full base64 image payload in conversation/tool history and resends it in later /v1/responses requests.
This causes unbounded context growth. In one observed case, a later request included multiple base64 image payloads in tool messages, producing an upstream Chat request of about 8.34 MB and about 5.7 million prompt tokens. Several OpenAI-compatible upstream providers then failed with 502/524 or HTTP 200 streams with no valid output.
Expected behavior
- User-submitted images can be included in the current request.
- After the image has been processed, Codex should not keep resending the full
data:image/*;base64,...payload as ordinary conversation context. - Large image payloads should be replaced by a compact reference, digest, placeholder, or omitted from subsequent context unless explicitly required again.
- Tool outputs containing large binary/base64 data should be aggressively pruned or summarized before being stored in long-lived conversation history.
Actual behavior observed
A later Codex request contained full base64 images inside historical tool messages. The largest tool message was a PNG data URL of about 7.68 million characters. Another JPEG data URL was about 426k characters.
The request shape after being converted to OpenAI Chat format was approximately:
messages = 77
roles:
system 4
user 9
assistant 39
tool 25
tools = 10
upstream_body ~= 8,343,294 bytes
prompt_tokens ~= 5,722,717
completion_tokens = 0
Largest historical message contents:
tool message[36] ~= 7,681,241 chars
[{"type":"input_image","image_url":"data:image/png;base64,..."}]
tool message[23] ~= 426,314 chars
[{"type":"input_image","image_url":"data:image/jpeg;base64,..."}]
This was not caused by the proxy duplicating request data. The actual upstream request body did not include the proxy diagnostic raw request field; the large payload was already present in the conversation messages submitted by Codex.
Failure mode
Observed downstream/upstream behavior included:
I-智枢云桥/gpt-5.5 -> 502 after about 203s
https://vsllm.com/gpt-5.5 -> HTTP 200 stream, first token after about 84.8s, then dropped
prompt_tokens = 5,722,717
completion_tokens = 0
Other OpenAI-compatible providers also failed for similar Codex /v1/responses requests with large historical context, usually as 502/524 or stream dropped/no valid output.
Why this matters
This makes long-running Codex sessions unstable after image/tool interactions:
- Context can grow by several MB from a single image.
- The same large image data can be resent repeatedly in later turns.
- Prompt token counts can explode into millions.
- Upstream providers may fail, timeout, or return incomplete streams.
- The failure is confusing because the user is not submitting a new image in the later turn.
Suggested fix
Codex should treat large image/base64 payloads as ephemeral request data rather than durable text/tool history.
Possible mitigations:
- Strip or replace
data:image/*;base64,...payloads from historical messages after the turn that consumes them. - Store a compact placeholder such as
[image omitted: sha256=..., bytes=...]in later context. - Apply max-size limits to tool outputs before persisting them into conversation history.
- Avoid serializing image payloads into
toolmessages that are replayed to the model in subsequent requests. - Add diagnostics/warnings when a request context contains very large base64 payloads.
The key point: submitting an image in the current user turn is fine, but resending the full base64 image in all later turns should be avoided.
Activity
github-actions commentedon Jun 15, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
wang1970 commentedon Jun 15, 2026
Thanks for the duplicate detection. I reviewed #28150. This issue is related, but I do not think it is an exact duplicate.
#28150 focuses mainly on Codex Desktop thread persistence/recovery problems: large rollout JSONL, screenshots/appshots/local_images poisoning compaction, Electron crash loops, and safe-resume/repair-thread behavior.
This issue focuses on a different downstream effect of the same underlying image/tool-output lifecycle problem: large historical image payloads are being replayed into subsequent model requests. In the observed case, a previous image/tool result was resent inside later
/v1/responsescontext asdata:image/*;base64,..., producing an actual upstream request of about 8.34 MB and about 5.7 million prompt tokens. That caused provider-side 502/524 failures or HTTP 200 streams with no valid output.So the overlap is:
The distinction is:
I think keeping this open is useful because it provides concrete evidence that the problem affects not only local session recovery, but also the actual request payload sent to the model in later turns.
wgu9 commentedon Jun 15, 2026
I checked current
mainfor the request-growth path described here. The issue still looks plausible from the current code, and the risky boundary seems to be long-lived history/replay ofFunctionCallOutputContentItem::InputImage, not logging.Relevant code paths I found:
codex-rs/core/src/tools/handlers/view_image.rs:ViewImageOutput::log_preview()correctly omits the data URL for logs, butto_response_item()returnsFunctionCallOutputContentItem::InputImage { image_url: self.image_url.clone(), ... }. So the model-visible response item still carries the full data URL.codex-rs/core/src/session/mod.rs:record_conversation_items()records the response items into history and persists the same items to the rollout before sending raw response items.codex-rs/utils/output-truncation/src/lib.rs:truncate_function_output_items_with_policy()applies the byte/token budget to text items, butInputImage { image_url, detail }is pushed through unchanged.formatted_truncate_text_content_items_with_policy()also preserves images after text truncation.codex-rs/core/src/context_manager/history.rs:for_prompt()only strips images when the target model does not support image input. For an image-capable target model, historical image content can remain model-visible.codex-rs/core/src/session/turn.rs:replace_last_turn_images("Invalid image")exists, but it is only used as an error recovery path forCodexErr::InvalidImageRequest, not as normal post-turn history pruning.codex-rs/core/src/session/mod.rs: rollout reconstruction can callprepare_response_items()only behindFeature::ResizeAllImages, and the comment explicitly says persisted rollouts are kept unchanged. That does not appear to provide an unconditional sanitizer for historicaldata:image/*;base64,...payloads.So I think the minimal fix should be separate from log redaction and separate from current-turn image submission. The safer owner boundary is the durable-history / replay path: after an image has been consumed for the current turn, historical tool-output images should be replaced with a compact placeholder or digest before they can be persisted/replayed into later prompts. Current-turn user images can still be sent normally.
Suggested regression test shape:
FunctionCallOutputcontent item withInputImage { image_url: "data:image/png;base64,...large..." }.data:image/and instead contain a bounded placeholder/reference.This is high impact because a single historical image can bypass the existing output truncation budget and be resent repeatedly, which matches the multi-MB / million-token request shape reported above.
100yenadmin commentedon Jun 18, 2026
I published a diagnostics branch that should help quantify this class of image/tool-output replay without changing persisted history yet:
Compare: https://github.com/openai/codex/compare/main...100yenadmin:codex/resume-payload-diagnostics?expand=1
Commit:
0e980ef9588d6dfb3619f31b0a112c193660451aI could not open the upstream PR directly because GitHub returned
GraphQL: 100yenadmin does not have the correct permissions to execute CreatePullRequest, but the branch is pushed to100yenadmin/codex.Scope note: this is not the durable-history sanitizer proposed above. It intentionally does not mutate persisted rollout/history or strip model-visible image payloads. It only logs non-content resume payload diagnostics for cold/running
thread/resume: turn/item counts, estimated item payload bytes, item class distribution, user-image/image-item counts, MCP/dynamic-tool output counts, command-output counts, and token-usage replay timing.Why it is relevant here: the active Desktop lockups in #28867 now include small active threads, so image-heavy/full-history replay is a stress signal rather than a proven sole cause. These diagnostics should let maintainers correlate future stalls with whether the resume payload actually contains image/tool-output heavy history, and then decide whether the next guardrail belongs in resume transport, Desktop rendering, token-usage replay, or durable-history image redaction.
100yenadmin commentedon Jun 18, 2026
@tibo-openai
Necmttn commentedon Jun 21, 2026
After first use, large image payloads should be replaced by digest-backed references in replay history.
Persist media type, byte count, hash, producer turn/tool id, and an optional retained blob pointer. Later context assembly can include a compact placeholder plus token/byte diagnostics, instead of resending base64 bodies that already served their turn.
Generated with ax.
aweiu commentedon Jul 1, 2026
any update?
hertzhzzz commentedon Jul 15, 2026
Repro / confirmation:
view_image→ persistent base64 in session → later turns 400context_length_exceededEnvironment
wire_api = "responses", basehttp://127.0.0.1:15721/v1gpt-5.6-sol,model_reasoning_effort = "xhigh"gpt-5.6-solascontext_window = 128000/max_context_window = 128000What happens
view_image(and/or browser screenshots).data:image/...;base64,...under image / tool-related items (input_imagecounts grow).User-facing (via local proxy) often becomes:
local proxy failed while handling Codex endpoint /responses+ the same cause string.Measured (redacted)
view_image/input_imageinput_image=26with 135data:imagemarkers in one sessionThis matches the core claim of this issue: images are expected on the turn they are viewed, but full base64 tool/image payloads appear to remain in history and get re-sent, causing unbounded context growth and permanent thread death.
Cross-links
Ask / expected behavior
/compactshould strip or externalize inline images; otherwise compact itself 400s.I can share additional redacted counters (image count vs body_bytes over time) if helpful. No secrets/payloads attached.
Cooper-X-Oak commentedon Jul 28, 2026
Windows Desktop reproduction: resumed image-heavy history causes later text-only turns to fail with HTTP 403
I can reproduce the same underlying replay problem on Windows with Codex Desktop.
Observed behavior
data:image/png;base64,...values.continuethen consistently fails withunexpected status 403 Forbidden.Sanitized local measurements
data:image/png;base64,...payloads.Expected behavior
Historical tool-output images should not be replayed as raw data URLs on later text-only turns. They should be pruned, bounded, externalized, or replaced with a compact reference/placeholder before durable history is replayed. A poisoned image-heavy thread should also have a supported repair path rather than causing every subsequent short request to fail.
No raw images, prompts, paths, session IDs details are included here for privacy.
suharvest commentedon Aug 1, 2026
Additional data point: this replay costs ~55 GB of upstream traffic in a single working day
The reports above focus on functional failures (
context_length_exceeded, 403/502). The same replay path also has a large and easily quantified network cost, which I don't think has been measured yet.Environment
codex-cli 0.146.0-alpha.9.2atChatGPT.app/Contents/Resources/codex; standalonecodex-cli 0.145.0also installedcodex(the Electron/app processes are accounted separately)One working day (2026-07-31), process
codexchatgpt.com)An upload/download ratio of roughly 35:1.
Per-request upload size
Across the 2,212 requests above 1 MB: median 13.3 MB, max 211 MB.
Two observations pointing at replay rather than organic context growth
Resumed sessions start the day already at full payload. The first requests of the working day (07:00) had a median upload of 38.2 MB with no ramp-up whatsoever. A session resumed from history immediately pays for everything accumulated on previous days.
Cost is strictly per-turn, and images dominate the session. Scanning rollout JSONL files for inline
data:image/*;base64,payloads:The average retained image is ≈1 MB, which lines up with the 30–50 MB request bucket at roughly 30–40 retained images per request.
Overnight, with those same session files on disk but no prompts submitted,
codexmade 87 connections and uploaded 0.22 MB over 7 hours. So there is no background cost — it is entirely per-turn replay.Code path
Consistent with the analysis @wgu9 posted earlier. In
codex-rs/utils/output-truncation/src/lib.rs,truncate_function_output_items_with_policy():InputTextcomputes a cost, decrementsremaining_budget, and falls back totruncate_textwhen over budgetInputAudiocomputes a cost viaestimate_audio_token_countand checks it against the budgetInputImageis pushed withimage_url.clone()unconditionally — no cost computed, no budget decremented, no truncationformatted_truncate_text_content_items_with_policy()behaves the same way: images are filtered out of the text merge and then re-appended verbatim. Callers includecodex-rs/core/src/context_manager/history.rs.A practical consequence:
tool_output_token_limitcannot bound this today, because images never participate in the truncation policy. That may be worth documenting even ahead of a fix, since it is a natural thing for users to reach for.InputAudioin the same match arm is a ready-made template for whichever bound is chosen (budgeted retention, or keeping the last N images and replacing older ones with a placeholder as @Necmttn suggested).Happy to re-measure with a patch applied if that would be useful.
bigbugneil commentedon Aug 26, 2026
Additional macOS Desktop evidence: direct
view_imagetool outputs dominate session storageI can confirm the same underlying issue on a current macOS Codex Desktop installation. In this case, the largest source of persistent image data is ordinary image-viewing tool output, not compaction alone.
Environment
26.818.61809(build7019)0.149.015.5(24F74), Apple Siliconview_image, including contact sheets, QA images, and screenshotsPrivacy-safe store-level measurements
A streaming scan was performed without printing or uploading prompts, image contents, project paths, credentials, or raw JSONL records.
1,247~/.codex/sessionssize:16.076 GiBdata:image/*;base64payload occurrences:14,68311.307 GiB70.3%of the entire session store36594 MiB502 MiBconsisted of Base64 image payloadsBreakdown of the 11.307 GiB of image payloads:
custom_tool_call_output:9.073 GiB—80.2%2.049 GiB—18.1%0.157 GiB—1.4%0.028 GiB—0.2%Representative persisted structure:
{ "type": "response_item", "payload": { "type": "custom_tool_call_output", "output": [ { "type": "input_image", "image_url": "data:image/png;base64,<redacted>", "detail": "original" } ] } }Repeatedly viewing the same local image creates another full inline payload. A clipboard screenshot of approximately 304 KB, for example, produced approximately 396 KiB of Base64 data in the rollout, consistent with Base64 expansion.
Why this is distinct and important
Existing reports correctly identify compaction and full-history forks as major amplification paths. This dataset shows that even without compaction being the dominant contributor, normal
view_imageusage can steadily grow the persistent store because full image bytes are written into ordinary tool-output records.This makes image-heavy review workflows accumulate many gigabytes even when the original images already exist as local files.
Requested product behavior
Related storage reports include #23257, #34337, and #40111. The current experimental cold-rollout compression path may reduce some physical storage, but compression alone does not address repeated tool-output persistence, content deduplication, active sessions, image discoverability, or safe garbage collection.
No raw rollout files are attached because they contain private screenshots, prompts, local paths, and project information.
HyperHe-dev commentedon Aug 28, 2026
macOS confirmation: sharp post-tool latency cliff after active
view_imagehistory reached ~90 MBI can add a current macOS Desktop reproduction where ordinary
view_imagetool outputs correlate with a severe per-model-step latency cliff on the official OpenAI backend.Environment
26.6.1, Apple Silicon (arm64)26.825.31414(build7287)codex-cli 0.150.0-alpha.12.2gpt-5.6-solmaxservice_tier: priorityThe separately installed
codex-cli 0.150.1was not used by this Desktop task.Privacy-safe rollout measurements
At the captured point:
110.4 MBcustom_tool_call_output:31input_image.image_urlcharacters:109.3 MB10view_imageresult:19.2 MBacross four images0258,400tokensNo prompts, image contents, local paths, credentials, raw rollout records, or task IDs are included here.
Latency transition
Before the active inline-image history reached approximately
72.6 MB, observed post-tool model-boundary delays were generally16–48 seconds.A later
view_imageresult added17.5 MBacross four images, bringing the active retained image history to approximately90.2 MB. The next model event arrived474 secondslater.After that point, repeated model-boundary gaps were:
Another four-image
view_imageresult added19.2 MB, bringing active retained image history to approximately109.3 MB. The next observed gaps were276sand269s.The latency cliff began at approximately
158,227 / 258,400input tokens, well before the nominal context limit. The 474-second gap followed a request reporting approximately99%cached input; most later long gaps also reported roughly93.6–99.3%cached input. A prompt-cache miss is therefore not required for the slowdown.In the captured interval, local tool calls themselves completed in at most 15 seconds. The multi-minute gaps occurred after tool output and
token_countwere persisted, before the next reasoning/tool/final-response event. Desktop logs showed no approval wait, rate-limit event, timeout, or stream error for those gaps.Historical controls on the same machine/account
gpt-5.6-sol+maxsession using bundled CLI0.150.0-alpha.8reached231,409input tokens, compacted six times, and had a maximum comparable post-tool boundary gap of16sabove 150k input tokens.53–189 MBof inline images over their full append-only history, but compaction kept the maximum active image payload between checkpoints to approximately23–37 MB. None showed a180s+comparable boundary gap above 150k input tokens.This does not prove the exact request body resent to the service; request-level telemetry would be needed for that. It does show a strong correlation between un-compacted inline
view_imagepayload growth and a sudden, persistent latency increase that ordinary token accounting did not predict.Suggested diagnostics/fix
InputImagebyte cost in history truncation and compaction thresholds.view_imagebytes and retain lightweight references instead of replayable data URLs.A separate UI symptom occurred in the same period: the Desktop log emitted 1,222
ResizeObserver loop completed with undelivered notificationserrors in roughly one hour. That may explain additional renderer sluggishness and appears related to #33996, but it does not explain the multi-minute absence of new rollout events by itself.chipfighter commentedon Sep 28, 2026
Adding a data point and a stopgap. On the Windows desktop app, I was iterating on image generation with GPT in one long task. After a dozen or so images, every request carried all of them again and had grown to about 44 MB. The connection kept dropping, the retries every few minutes never went through, and the only way out was a new task, which loses the conversation.
Until Codex does this itself, I built an open-source plugin that does it by hand (disclosure: I'm the author): Codex Attachment Manager. A tab in the desktop app's side panel lists every image in the task. You uncheck the ones you no longer need, and from the next message on a local proxy replaces them with short placeholders before the request goes out. Same task, same conversation; the rollout on disk isn't modified.
In case it helps the built-in version, two things mattered in my testing:
need IMG-xxxwhen they need the original.Caveat for the original report: the plugin only works with the default OpenAI backend. It won't install over your own
openai_base_url, so it doesn't help with OpenAI-compatible providers. It's also desktop-only, and results of Codex's built-in image generation can't be unchecked. Tested on Windows with Codex 26.924; macOS and Linux only pass CI so far.xiao-do-do commentedon Oct 2, 2026
Another affected macOS user here. This is particularly disruptive during multi-round image generation: old image payloads accumulate, reconnect attempts reach 5/5, and HTTP/SSE fallback is followed by repeated large image uploads that greatly reduce conversation efficiency.
I am also affected on macOS while iterating on image generation across multiple turns. After reconnect attempts reach 5/5, the transport falls back to HTTP/SSE. Continuing the same conversation then causes repeated large image uploads, heavy bandwidth use, and substantial delays between turns. This severely affects Codex conversation efficiency and makes iterative design work difficult, including ordinary follow-up messages after image generation.
Please correlate this with #38846 (WebSocket close code 1009 / message too big), #24550 (large inline-image history causing WebSocket fallback), and #23665 (recovery from HTTP fallback). A 1009 close and the subsequent HTTP/SSE fallback are different stages of the failure: an oversized WebSocket message can trigger the disconnect, and fallback can then expose the repeated full-history upload cost. I have not captured the close code from my own session, so I am reporting the retry/fallback and bandwidth symptoms rather than claiming a confirmed 1009 reproduction.
Please treat historical-image byte size as a separate constraint from model token usage. Stable file references or bounded image retention would prevent small follow-up turns from repeatedly transferring large historical payloads. Please also provide a recovery action that preserves the text conversation, since starting over repeatedly interrupts design work.
lyouroboros109 commentedon Oct 12, 2026
Adding feedback about the repeated historical-image uploads described in this issue and #43015. Old images reportedly get retransmitted on later requests, including text-only turns that do not need those images. The resulting bandwidth usage and waiting time make extended conversations difficult to use.
Could maintainers please confirm the current status and the responsible team? If a fix has shipped, please identify the versions that include it. If it remains unresolved, please provide an official workaround and a supported way to remove historical images from an existing conversation while preserving its useful text context.
Please also consider bounding aggregate image/request bytes and replacing unnecessary historical image payloads with references, so small follow-up requests do not repeatedly pay the full upload cost.
This comment conveys user feedback; it is not a new locally measured reproduction. No private images or session logs are attached.