"% context left" divides by a hardcoded 12,000-token baseline, so it's wrong in both directions #40756

What version of Codex CLI is running?

codex-cli 0.149.0

What subscription do you have?

ChatGPT Business

Which model were you using?

gpt-5.4 (model-independent — the defect is in the display arithmetic)

What platform is your computer?

Darwin 25.5.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

No response

Codex doctor report

What issue are you seeing?

The context indicator divides by a hardcoded constant standing in for the fixed cost of a request, so the percentage is off for basically every session — and off in both directions depending on how much tooling you have loaded.

TokenUsage::percent_of_context_window_remaining does this:

effective_window = context_window - BASELINE_TOKENS
used             = max(tokens_in_context_window - BASELINE_TOKENS, 0)
remaining%       = (effective_window - used) / effective_window

const BASELINE_TOKENS: i64 = 12000 lives at codex-rs/protocol/src/protocol.rs:2241, with a second copy at codex-rs/tui/src/token_usage.rs:9.

The subtraction cancels in the numerator — it reduces to context_window - tokens_in_context_window — so the baseline only ever moves the denominator. It never washes out, and whatever it's wrong by lands straight on the displayed number.

It's too big for a plain session. This repo's own test config is a fair reference point: suite::client::token_count_includes_rate_limits_snapshot sends 8 tools, and pricing that request's fixed parts with Codex's own approx_token_count comes to 7,015 tokens — 5,337 of base instructions and 1,678 of tool schemas. Subtracting 12,000 from the window shrinks the denominator, so the indicator claims more headroom than actually exists.

And it's too small once you load things. Enough MCP servers, plugins and skills and the real fixed cost passes 12,000, the sign flips, and the indicator under-reports instead. There's no configuration where 12,000 happens to be right except by luck.

The comment on the constant is also wrong on its last clause. It reads "Includes prompts, tools and space to call compact", but auto-compaction reserves its own headroom over in codex-rs/core/src/session/context_window.rs, comparing sess.get_total_token_usage() against model_info.auto_compact_token_limit() plus TokenBudgetConfig::fallback_buffer_tokens. Nothing in the compaction path reads BASELINE_TOKENS at all.

The good news is that the constant is display-only — every read of it sits inside the matching percent_of_context_window_remaining — so fixing it doesn't have to touch anything that affects behaviour.

What steps can reproduce the bug?

The arithmetic is deterministic, so this is more inspection than timing:

1. Start a session with no MCP servers or plugins and note the reading. The denominator is context_window - 12000, while the real fixed cost on the default configuration prices at 7,015 — so it's optimistic by construction.
2. Add enough MCP servers and skills to clearly push the fixed context past 12,000 tokens, restart, and look again. Now the formula subtracts less than the request actually costs, and the reading goes pessimistic.
3. For the number behind step 1: codex-rs/core/tests/suite/client.rs::token_count_includes_rate_limits_snapshot is the request whose fixed parts price at 5,337 + 1,678.

What is the expected behavior?

Divide by what the request actually costs — measured off the Prompt about to be sent: base instructions, tool payload, output schema — instead of by a constant.

No tokenizer required. Codex already ships approx_token_count (utils/string/src/truncate.rs) and deliberately uses that byte-density heuristic for history estimation and truncation, so pricing the baseline the same way keeps everything on one accounting.

Additional information

Three things that are easy to get wrong here, from having implemented it:

The tool payload has two wire sizes. build_responses_request collapses every spec into a single namespace object when model_info.use_responses_lite && provider.capabilities().namespace_tools, and sends them one by one otherwise — so you have to price whichever branch the request will actually take.

Leave estimate_token_count_with_base_instructions alone. It omits tool schemas, which looks like the same bug, but it feeds recompute_token_usage → get_total_token_usage → token_limit_reached. Changing it moves when Codex compacts, which is what turns a display fix into a behaviour change.

Once the baseline is per-request, it has to belong to the request whose usage is on screen. Local compaction builds a Prompt with no tools and no output schema (core/src/compact.rs) but still reports provider usage, so carrying the previous turn's baseline onto that event subtracts a fixed cost that request never paid. It shows up as free space jumping right after a compact.

I have a branch doing only this, with the measurement floored at 12,000 so nothing changes for sessions below it: iamenahs#1. Happy for it to be taken directly or just used as a reference — I know this repo doesn't take external pull requests.

Related: #27898 proposes a /context breakdown card, and @alainkaiser's feat/context-breakdown implements it. That branch measures the same categories this needs, but renames the TUI constant rather than replacing it — so the card's own "Status line" row is still computed against 12,000 while every row beside it is measured.