Automatic compaction double-counts prior reasoning against reported token usage and triggers unnecessary compaction #49026

What version of Codex CLI is running?

Maintained Codex fork based on upstream commit 12de0e3

What subscription do you have?

ChatGPT Pro — $200/month

Which model were you using?

GPT-6 Astra Ultra

What platform is your computer?

Windows 10 x64

What terminal emulator and version are you using (if applicable)?

No response

Codex doctor report

What issue are you seeing?

Summary

Codex prematurely triggered automatic compaction because its context estimator added 72,772 estimated tokens for prior encrypted reasoning on top of a completed response's 177,413 reported tokens.

The resulting count was 250,193 tokens, crossing the 244,800 automatic-compaction threshold, even though the last reported usage remained 67,387 tokens below that threshold.

The incorrect threshold decision then caused a real additional remote compaction request consuming:

• 177,445 input tokens
• 176,768 cached input tokens (already included in input)
• 4,091 output tokens
• 181,536 total reported tokens

There was no manual compaction request.

Exact calculation

Immediately before compaction:

The decision was:

177,413 + 72,772 + 8 = 250,193

250,193 >= 244,800  -> automatic compaction
250,193 < 258,400   -> full context-window limit not reached

Reconstructing the recorded history reproduces the 250,193-token count exactly.

Root cause

The additional 72,772 tokens came from 148 encrypted reasoning items from earlier model responses retained in history.

When server_reasoning_included is false, Codex effectively calculates:

last reported total
+ estimated earlier reasoning
+ newly added items

The affected WebSocket handshake did not contain:

X-Reasoning-Included

so this branch was used.

The defect is that a historical subset is added to an already completed request's reported usage without establishing that the two quantities are disjoint.

The 148 reasoning items were not newly generated reasoning. They matched the archived preceding request input by hash.

Actual remote operation caused by the bug

The compaction began immediately after the incorrect threshold decision.

This was not a UI-only accounting error. The incorrect context calculation directly caused an additional remote model operation.

Upstream code

The running fork is based on upstream commit:

12de0e395d3313bc564190d983cb4f5acf0be713

These functions were compared directly with that upstream commit and are unchanged:

codex-rs/core/src/context_manager/history.rs

get_non_last_reasoning_items_tokens
get_total_token_usage
is_user_turn_boundary

Relevant upstream code:

core/src/context_manager/history.rs
    prior-reasoning estimate and total-token calculation

core/src/state/session.rs
    server_reasoning_included state and history accounting

core/src/session/context_window.rs
    automatic-compaction threshold comparison

core/src/session/turn.rs
    server flag handling and compaction decision

codex-api/src/sse/responses.rs
    X-Reasoning-Included handling

codex-api/src/endpoint/responses_websocket.rs
    corresponding WebSocket handling

Correct accounting

For a settled request with a complete request-bound estimate, the context baseline should compare complete quantities rather than add historical reasoning to an already completed total:

baseline = max(
    last server-reported input + output,
    estimated complete composed input + actual output
)

next context estimate =
    baseline + genuinely new local-item estimate

For this incident:

archived complete-input estimate: 201,228
developer-message normalization:    -990
inserted text upper bound:          +2,812
                                     -------
complete composed-input bound:     203,050

Therefore:

max(177,413, 203,050 + 509) + 8
= 203,567

203,567 < 244,800

The correct decision does not trigger automatic compaction.

Related open issues

This sits within a much larger existing set of open Codex usage/accounting defects:

• #41220 — Abnormal Codex usage/quota depletion and usage-accounting inconsistencies — cross-report tracker
• #47825 — Hosted search usage causes premature auto-compaction
• #47003 — Auto-compaction request usage is never added to total_token_usage
• #45074 — Automatic compaction at 388K displayed tokens uses a 623K internal count
• #48919 — 36,318-token request followed by ~6% 5-hour depletion and ~2.95M daily tokens
• #46681 — Weekly usage dropped from ~75% to 16% in 19 minutes despite 98,700 reported tokens
• #44386 — ~150K cached input replayed per tool call, rapidly draining usage
• #38495 — Full-context polling loop consumed 34.6M tokens after work had completed
• #35463 — Subagent history/accounting behavior associated with an entire weekly quota drain
• #32640 — Repeated wait/resampling behavior causing massive token consumption

Customer impact

I pay $200/month for ChatGPT Pro, and I now maintain both my own Codex fork and an increasingly substantial local instrumentation and patching layer around it.

I find the current situation completely absurd: I am spending a meaningful portion of my paid OpenAI allowance using OpenAI's own models to identify, diagnose, and patch defects in OpenAI's own Codex client that themselves waste that allowance.

This bug is a concrete example. An incorrect client-side accounting decision caused an additional 181,536-token remote operation that should not have occurred at that point.

This is also not an isolated class of problem. The issue tracker already contains a substantial number of open reports involving incorrect accounting, unnecessary context replay, runaway polling, premature compaction, background activity, and subscription usage that does not reconcile with observable work.

A normal client bug wastes time.

A client bug that causes unnecessary inference consumes the product the customer paid for.

At $200/month, I should not have to spend an increasing portion of that same finite allowance maintaining a private patch layer around the official client simply to prevent avoidable usage consumption.

What steps can reproduce the bug?

1. Use a Codex session containing retained encrypted reasoning items from earlier model responses.

2. Reach a completed response where the server-reported token total remains below the automatic-compaction threshold.

3. Have server_reasoning_included == false, as occurs when the response does not contain X-Reasoning-Included.

4. Append a new history item after that completed response.

5. Allow the normal Codex history-accounting path to evaluate the automatic-compaction threshold.

In the captured incident:

last reported total:               177,413
estimated prior reasoning:          72,772
new-item estimate:                       8
                                     -------
calculated context count:          250,193

automatic-compaction threshold:    244,800
context-window limit:              258,400

Codex therefore triggers automatic compaction even though the last reported total remains 67,387 tokens below the threshold.

The 72,772-token component is produced from the 148 retained encrypted reasoning items selected by the history estimator.

Reconstructing the recorded history reproduces the final 250,193-token count exactly.

The resulting automatic compaction produces an additional remote request:

input_tokens:          177,445
cached_input_tokens:   176,768
output_tokens:           4,091
total_tokens:          181,536

Relevant unchanged upstream functions:

codex-rs/core/src/context_manager/history.rs

get_non_last_reasoning_items_tokens
get_total_token_usage
is_user_turn_boundary

Upstream base:

12de0e395d3313bc564190d983cb4f5acf0be713

What is the expected behavior?

No response

Additional information

No response