Skip to content

Image-heavy multi-agent task replays inherited image context, causing 1.48B tokens, 70.73GB traffic, and heavy swap/disk growth #33235

Open

Description

@lgch3136

Environment

  • Codex Desktop: 26.707.72221 (build 5307)
  • Bundled Codex CLI: 0.144.2
  • OS: macOS 26.5.2 (25F84)
  • Hardware: Intel MacBook Pro (MacBookPro16,1), 32 GiB RAM
  • Workload: one long-running root task that produced 279 images through child tasks
  • Task tree: 1 root thread + 25 descendants (26 threads total)

All figures below are sanitized aggregates. I am not attaching raw rollout JSONL because it contains private prompts, image bytes, local paths, and tool outputs.

Summary

An image-heavy multi-agent task appears to repeatedly inherit and replay large inline image context across child threads and subsequent model turns. In this case, the same task family accumulated:

  • 1,480,645,018 total tokens in Codex local state
  • 70.73 GB of visible upstream traffic during the two-day task window
  • 2.916 GB of allocated rollout files for the 26-thread family
  • 0.968 GB of generated-image cache
  • 0.752 GB in a deleted root rollout still held open by the Codex process
  • 16.106 GB of new swap files created during the high-load period

This is related to existing image-heavy rollout and memory issues, but the additional signal here is the multiplicative effect across a multi-agent task tree, including token accounting and measured upstream network traffic while the task was running.

Token evidence scoped to this single task family

I did not sum global/account-wide usage. I recursively selected only the root thread and descendants through the local thread_spawn_edges relationship.

Local state reported:

threads:       26
total_tokens:  1,480,645,018
last_update:   2026-07-15 09:26:37 local time

The parseable rollout token records accounted for 1,476,473,221 tokens:

input_tokens:           1,471,956,553
cached_input_tokens:    1,399,287,552
uncached_input_tokens:     72,669,001
output_tokens:              4,516,668
cached share of input:          95.06%

cached_input_tokens is treated as a subset of input tokens, not added again.

Five high-traffic image child tasks alone recorded:

model turns:       4,244
total tokens:      719,129,229
share of family:        48.57%
cached share:           94.85%
average input/turn: 128,807–135,832 tokens

This suggests that prompt caching reduced uncached processing, but did not prevent the already-large context from being counted and repeatedly processed across thousands of turns.

Image-context evidence

The rollout records contain inline PNG data URLs/Base64 image payloads.

Observed child-task inheritance:

  • Three child tasks each started with 26 older images already present, approximately 72.1 MB of image payload per child.
  • They then generated 77, 92, and 100 new images respectively.
  • Two later retry tasks each started with 103 older images, approximately 269.1 MB before producing their small retry batch.
  • A local cumulative-payload estimate for five key image tasks exceeded 53 GB before protocol overhead, text context, root-thread activity, and retries.

The 53 GB figure is a local uncompressed-payload estimate, not a claim that it equals billed network bytes.

Network evidence

A third-party upstream traffic meter showed 70.73 GB of visible records during the two-day task window:

day 1 visible records: 45.00 GB
day 2 visible records: 25.73 GB
total visible:         70.73 GB

Records below 10 MB were omitted by that meter.

On day 2, image-task windows could be aligned precisely:

  • Three image children started at 00:50, 00:51, and 00:52.
  • Upstream usage from 00:52 to 03:45: 22.50 GB.
  • Two retry children started at 07:57 and 07:58.
  • Upstream usage from 07:59 to 08:27: 2.27 GB.
  • Combined time-aligned usage: 24.77 GB, or 96.3% of the visible day-2 total.
  • Large traffic records nearly disappeared while the image tasks were paused between approximately 04:00 and 06:00.

The traffic meter cannot attribute bytes to a Codex thread ID, so the network relationship is a time-correlation observation rather than per-thread byte accounting.

Local disk and memory evidence

Task-family Codex storage:

26 rollout files, allocated:       2,915,504,128 bytes
corresponding generated images:      968,327,168 bytes
deleted root rollout still open:     751,684,963 bytes
task-associated log payload:         123,500,431 bytes (8,908 rows)

The root rollout was scrubbed/slimmed from approximately 742 MB to approximately 10 MB after confirming images were saved. However, lsof +L1 showed the Codex process still holding the deleted old inode, so approximately 751.7 MB was not returned to the filesystem until the file descriptor/process closed.

Swap evidence:

APFS VM volume peak:       19,328,688,128 bytes
new swap files in window:  15 × 1 GiB
new swap allocation:       16,106,127,360 bytes
later VM volume usage:     12,567,490,560 bytes
free-space recovery:        6,821,634,048 bytes (about 6.82 GB)

The VM volume is system-wide, so I cannot assign every swap byte to one Codex thread. The creation times were concentrated in the image-task high-load window.

No local Time Machine snapshots or extra APFS Data snapshots were present.

Reproduction shape

  1. Start a long-running root task that needs hundreds of generated images.
  2. Let the root task spawn multiple image-producing child tasks.
  3. Allow child tasks to inherit the parent transcript containing prior inline image results.
  4. Continue generating images and making model/tool turns in each child.
  5. Spawn retry children from the already image-heavy root context.
  6. Observe:
    • cumulative token usage per thread,
    • rollout JSONL size,
    • repeated inline image_url/image payloads,
    • generated-image cache size,
    • upstream traffic,
    • process RSS and swap,
    • deleted-but-open rollout files after slimming.

Expected behavior

  • Generated images should be stored once and referenced from rollouts by path/content hash rather than duplicated as full inline Base64 whenever possible.
  • Child tasks should not inherit raw historical image bytes unless those images are explicitly needed.
  • Multi-agent image workflows should have rollout/context size gates and warnings.
  • Codex should expose configurable token, context, disk, or network budgets for long-running tasks.
  • Compaction/slimming should not leave large deleted rollout inodes retained indefinitely.
  • The app should warn before a thread/task family reaches dangerous rollout, memory, or cumulative-token levels.

Actual behavior

The task continued normally in the UI while token usage, upstream traffic, rollout storage, memory pressure, and swap grew substantially. Most input tokens were cache hits, but the aggregate context-processing count and local/network resource usage still became very large.

Related issues

This report is not primarily about reopening/rendering an existing large thread. The resource amplification was observed while the multi-agent image task was actively running.

Privacy / additional diagnostics

I can provide additional sanitized record counts, timestamps, SQL queries, and file-size summaries if useful. I cannot post the raw rollout or generated images publicly because they contain private user content.

Activity

  1. added
    bugSomething isn't working
    appIssues related to the Codex desktop app
    subagentIssues involving subagents or multi-agent features
    contextIssues related to context management (including compaction)
    on Jul 15, 2026
  2. dorukardahan commented on Jul 23, 2026

    @dorukardahan

    i have an independent reproduction from another long-running Codex 0.145.0 session.

    i counted real server usage updates after one affected compaction, not the provisional local recomputation. The first 53 updates reported:

    Metric Value
    Total input tokens 10,173,598
    Cached input tokens 9,472,768
    Uncached input tokens 700,830
    Median cached input share 94.9%

    The matching compacted checkpoint retained 32 high-detail inline images. The current estimator in Codex maps them to about 59,000 model-visible image tokens:

    /// Returns the same coarse, model-visible token estimate used for full history estimates.
    ///
    /// Ordinary items are JSON-serialized, so callers estimating many items should reuse these
    /// results instead of repeatedly estimating the full history.
    pub(crate) fn estimate_item_token_count(item: &ResponseItem) -> i64 {
    let model_visible_bytes = estimate_response_item_model_visible_bytes(item);
    approx_tokens_from_byte_count_i64(model_visible_bytes)
    }
    /// Approximate model-visible byte cost for one image input.
    ///
    /// The estimator later converts bytes to tokens using a 4-bytes/token heuristic
    /// with ceiling division, so 7,373 bytes maps to approximately 1,844 tokens.
    const RESIZED_IMAGE_BYTES_ESTIMATE: i64 = 7373;

    For a workspace on the current token-based Codex rate card, the GPT-5.6 Sol input rates are 125 credits per million uncached input tokens and 12.5 credits per million cached input tokens. This 53-update input mix maps to about 206 input credits before output:

    https://help.openai.com/en/articles/20001106-codex-rate-card

    This is not an account invoice and it does not isolate the image-only share. The usage event has no modality split. It does show that a high cached share does not make the repeated large context free.

    i did not inspect or upload prompts, paths, images, tool output, project names, thread IDs, or account-wide totals. A useful regression signal would be sanitized per-request retained image count and estimated image tokens, together with a test that a second compaction request does not replay images covered by the first compaction.

  3. qingye-lab commented on Jul 24, 2026

    @qingye-lab

    Additional sanitized macOS reproduction confirming the same network-amplification pattern, with a much smaller image count.

    Environment

    • Codex Desktop: 26.721.31836 (build 5828)
    • Bundled CLI: codex-cli 0.146.0-alpha.3.1
    • macOS 27.0 (build 26A5388g), Apple Silicon
    • Measurement window: 24 hours
    • No raw rollout, prompts, images, local paths, or thread IDs are included below

    Image/context evidence

    One long-running root thread received 30 PNG attachments in a single user message.

    Sanitized structural inspection of the rollout showed:

    • inline input_image data-URL payload: 44,102,256 bytes across 30 images
    • source image bytes before base64 expansion: approximately 33 MB
    • root rollout size: 948.5 MiB
    • root task: 34 turns, 3,367 tool calls
    • compaction records: 33
    • bytes in compaction records: 843.8 MiB
    • each large compacted checkpoint remained approximately 42.1 MiB and retained the same 30 images in replacement_history
    • root input tokens: 540,338,229
    • cached input tokens: 508,753,920 (94.15% of input)

    Several descendant tasks inherited the compacted image context. Three descendants that did not contain a new direct image message still produced 11–15 compacted records of roughly 42.1 MiB each. Their recorded input-token totals were approximately 488–490 million each, with similarly high cache-hit ratios.

    Network evidence

    I decoded the per-host/per-second history stored by a local Clash/Mihomo client and aggregated the same 24-hour window:

    Destination scope Upload Download Total
    OpenAI/ChatGPT domains 94.177 GB 1.743 GB 95.920 GB
    chatgpt.com alone 94.173 GB 1.570 GB 95.743 GB

    A separate per-process traffic display showed approximately 84 GB for Codex over its slightly different window.

    The timing aligned closely:

    • before the 30-image message, OpenAI upload was negligible
    • in the hour containing the image message, upload increased to 2.496 GB
    • subsequent active hours commonly uploaded 3–10 GB/hour
    • peak observed OpenAI upload was 10.452 GB in one hour
    • during a live sample, three concurrent chatgpt.com connections mapped to the Codex process had each uploaded approximately 33 MB

    The 44.1 MB inline image payload repeated roughly 2,100 times is sufficient to explain the measured upstream traffic. Prompt caching clearly reduced uncached model processing, but it did not prevent the client-side payload from being retransmitted.

    All chatgpt.com traffic in this case used configured proxy nodes rather than DIRECT, so the amplification also consumed metered proxy traffic.

    Why this additional repro may matter

    This case reached approximately 94 GB upstream in one day with only 30 user-attached images, rather than hundreds of generated images. It suggests the issue can be triggered by a relatively ordinary batch of screenshots when the parent task runs for a long time, performs many tool/model cycles, compacts repeatedly, and spawns descendants.

    The observed behavior is consistent with inline image bytes being retained in hot history and replayed across later turns, compactions, and inherited subagent context. A content-addressed attachment reference, explicit retained-image byte/count budget, and excluding historical image bytes from default child inheritance would likely prevent this amplification.

  4. bbingz commented on Jul 25, 2026

    @bbingz

    Analysis (community)

    Residual analysis for #33235 (image-heavy multi-agent tasks replaying inherited image context → extreme token/traffic cost) under docs/contributing.md — invitation-only. No unsolicited PR.

    Root-cause hypothesis (narrowed)

    Multi-agent V2 spawn defaults fork_turns to "all" → FullHistory, which re-copies user/system/developer Message items including ContentItem::InputImage. That is a real OSS amplification path for attachment-style images. It is not true that every image-bearing wire type is re-injected: tool-output images and ImageGenerationCall are explicitly dropped by the fork filter.

    Verified at HEAD 4c43465133:

    1. V2 default fork mode = FullHistory.
    codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs SpawnAgentArgs::fork_mode:

    • missing / empty fork_turns → .unwrap_or("all")
    • "all" → Some(SpawnAgentForkMode::FullHistory)
    • "none" → None (no parent history fork)
    • positive integer string → LastNTurns(n)
    • fork_context present → error (use fork_turns)

    So the default spawn path forks full parent rollout history unless the model opts out.

    2. What FullHistory keeps vs drops (keep_forked_rollout_item).
    codex-rs/core/src/agent/control/spawn.rs:

    • Keep: Message role system/developer/user (any content — including ContentItem::InputImage); assistant only if FinalAnswer; Compacted / EventMsg / SessionMeta; optional TurnContext/WorldState under full history.
    • Drop: FunctionCall / FunctionCallOutput / CustomToolCall* / ImageGenerationCall / reasoning / compaction triggers / etc.

    3. Image carriers (protocol) × inheritance.

    Carrier Survives FullHistory?
    User/dev Message + ContentItem::InputImage YES — primary residual
    Assistant FinalAnswer Message + images YES if FinalAnswer
    view_image → FunctionCallOutput + InputImage NO (FC output dropped)
    ImageGenerationCall NO

    So the reporter narrative “multi-agent replays inherited image context” is partially right (message-embedded attachments under default FullHistory, multiplied per child) and partially over-broad if read as “all image types including tool/ig payloads.”

    4. V1 contrast.
    Legacy multi_agents spawn uses FullHistory only when fork_context=true (opt-in). V2 inverted the default to full fork — important when attributing regressions.

    5. Distinct from #33493 / compaction (R-S).
    Compaction image-token / re-fire issues are a different axis from spawn-time message inheritance.

    Ranked fix outline (all OSS):

    1. Change V2 default fork_turns from all → none or a small N (behavior change — needs product call).
    2. In the Message keep path / keep_forked_rollout_item: strip or stub ContentItem::InputImage unless opt-in.
    3. Document fork_turns defaults and image inheritance in the spawn tool schema description.
    4. Telemetry: count InputImage bytes/items copied at fork time.

    Fail-first tests: (1) default spawn without fork_turns → FullHistory; (2) parent user message with InputImage appears in child history; (3) parent ImageGenerationCall / FC-output image does not; (4) fork_turns=none → no parent response history.

    Happy to help sketch the strip-images-on-fork patch if maintainers want the safer default. No unsolicited PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    appIssues related to the Codex desktop appbugSomething isn't workingcontextIssues related to context management (including compaction)imagenperformancesubagentIssues involving subagents or multi-agent features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions