Pre-turn compaction clears reasoning-included state before accounting #32483

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   contextIssues related to context management (including compaction)Issues related to context management (including compaction)

   on Jul 11, 2026
2. shtratos commented on Jul 11, 2026

   

   shtratos

   on Jul 11, 2026

   Author

   More actions

   Follow-up: cold resume had the same accounting gap because
   server_reasoning_included was previously in-memory only.

   I added a second regression:

   auto_compact_restores_reasoning_header_after_resume
   

   It runs two reasoning responses with X-Reasoning-Included: true, shuts down,
   resumes from the persisted rollout, and submits a third turn. Before the
   follow-up change, the test failed twice because resume defaulted the bit to
   false and issued an unwanted compact request.

   The branch now persists this bit as a dedicated rollout-only
   ServerReasoningIncluded(bool) marker and restores the latest marker on
   resume/fork. The marker is deliberately ignored by thread history, search,
   memory export, and app-server materialization; it does not change
   TokenCountEvent or other client-visible event JSON.

   Backward compatibility:

   • old rollouts have no marker and retain the existing conservative false
     default;
   • after the first successful response under the patched runtime, a marker is
     persisted and future resumes restore the exact accounting contract.

   Current branch:

   https://github.com/shtratos/codex/commits/fix/preserve-reasoning-header-preturn

   New commit:

   shtratos@0da44b4a6

   Fresh verification on current main:

   auto_compact_restores_reasoning_header_after_resume ... ok
   auto_compact_preserves_reasoning_header_for_next_pre_turn_check ... ok
   auto_compact_runs_when_reasoning_header_clears_between_turns ... ok
   auto_compact_counts_encrypted_reasoning_before_last_user ... ok
   just test -p codex-protocol ... passed
   just test -p codex-rollout ... passed
   just fix -p codex-core -p codex-protocol -p codex-rollout \
     -p codex-thread-store -p codex-state -p codex-app-server-protocol \
     -p codex-app-server ... passed
   just fmt ... passed
   

   The branch is ready for an invited PR if this approach matches the intended
   fix.

   👍React with 👍1Reacted by Hu Yao
3. coinman-dev commented on Oct 4, 2026

   

   coinman-dev

   on Oct 4, 2026

   More actions

   Same cause here on Codex 0.159.2 with gpt-6.1-sol under WSL2. A small client patch that stops the double count is described in #39767 (comment): with it the same chat went from being compacted at 734,065 tokens to 922,856 with no early compaction, and its regression test fails on unpatched 0.160.0. Patch and notes: https://github.com/coinman-dev/codex-context-accounting-fix
4. ghzlatarev commented on Oct 12, 2026

   

   ghzlatarev

   on Oct 12, 2026

   More actions

   Still reproducible on stock 0.162.1 (macOS arm64) and an unmodified source build at 7c9bfb910ef65c8f4a957c53affbd43d5fac2882. This adds a binary-level, credential-free app-server reproduction to the source-level test in the issue.

   The fixture uses a fresh temporary HOME/CODEX_HOME/workspace and a localhost Responses SSE mock. It submits three turns. Every accepted response reports input=100, output=10, total=110. The first response optionally contains a 120,000-character opaque synthetic reasoning item (A repeated; not real encrypted model data). Post-turn compaction is disabled; the pre-turn threshold is 20,000.

   Binary / fixture	Header on every response	Limit	Compactions	Requests	Turns
   0.162.1, reasoning item	true	20,000	1	4	3 completed
   source build, reasoning item	true	20,000	1	4	3 completed
   0.162.1, reasoning item	absent	20,000	1	4	3 completed
   0.162.1, reasoning item	false	20,000	1	4	3 completed
   0.162.1, no reasoning item	absent	20,000	0	3	3 completed
   0.162.1, reasoning item	true	30,000	0	3	3 completed

   In the true case, request 3 is local compaction (tools: []), despite the previous response explicitly asserting that its usage includes replayed reasoning. The no-reasoning and higher-limit controls isolate the accounting effect. The mock deliberately gives a small reported usage relative to the opaque payload so the client-side estimate becomes observable; these numbers are a controlled contract fixture, not production token measurements.

   The tested source still clears the flag at regular-task startup, before pre-turn compaction. Accounting then adds older reasoning estimates when the flag is false. tasks/regular.rs is byte-identical on inspected main 7e9c061ceb388c67097acac069d6b7f8a60e5011; that newer commit was not rebuilt.

   The fix direction already proposed here—preserve the previous response's flag through pre-turn accounting, then clear it before the upcoming response—is consistent with these observations. This probe does not test cold resume or the proposed patch.

   Run the standard-library Python script below:

   python3 compaction-accounting.py --binary codex --reasoning --header true
   python3 compaction-accounting.py --binary codex
   python3 compaction-accounting.py --binary codex --reasoning --header true --limit 30000

   The script prints structural evidence and turn outcomes. Exit 0 means the probe completed, not that Codex passed a regression assertion. Temporary fixture data is retained under the printed directory for inspection.

   Standalone reproduction

   #!/usr/bin/env python3
   """Credential-free stock Codex compaction accounting/request-shape probe."""
   import argparse,http.server,json,os,queue,subprocess,tempfile,threading,time
   from pathlib import Path
   
   def main():
    ap=argparse.ArgumentParser();ap.add_argument('--binary',required=True);ap.add_argument('--header',choices=['absent','true','false'],default='absent');ap.add_argument('--reasoning',action='store_true');ap.add_argument('--strict-empty-tools',action='store_true');ap.add_argument('--limit',type=int,default=20000);a=ap.parse_args()
    root=Path(tempfile.mkdtemp(prefix='codex-compact-',dir='/tmp'));home=root/'state';cwd=root/'workspace';home.mkdir();cwd.mkdir();requests=[];notices=[]
    class Mock(http.server.BaseHTTPRequestHandler):
     def log_message(self,*args):pass
     def do_POST(self):
      b=json.loads(self.rfile.read(int(self.headers.get('Content-Length',0))));requests.append(b);ix=len(requests);tools=b.get('tools');compact=isinstance(tools,list) and not tools
      if a.strict_empty_tools and compact and b.get('tool_choice') not in [None,'none']:
       self.send_response(400);self.send_header('Content-Type','application/json');self.end_headers();self.wfile.write(b'{"error":{"message":"Synthetic strict endpoint: tool_choice set with no tools","type":"invalid_request_error"}}');return
      items=[]
      if a.reasoning and ix==1:items.append({'id':'rs_probe','type':'reasoning','summary':[],'encrypted_content':'A'*120000})
      items.append({'id':'msg_'+str(ix),'type':'message','role':'assistant','content':[{'type':'output_text','text':'Synthetic summary.' if compact else 'Synthetic reply.','annotations':[]}]})
      events=[{'type':'response.created','response':{'id':'resp_'+str(ix)}}]
      for n,item in enumerate(items):events += [{'type':'response.output_item.added','output_index':n,'item':item},{'type':'response.output_item.done','output_index':n,'item':item}]
      events.append({'type':'response.completed','response':{'id':'resp_'+str(ix),'status':'completed','output':items,'usage':{'input_tokens':100,'output_tokens':10,'total_tokens':110,'input_tokens_details':{'cached_tokens':0},'output_tokens_details':{'reasoning_tokens':5 if a.reasoning and ix==1 else 0}}}})
      data=''.join('event: '+e['type']+'\ndata: '+json.dumps(e)+'\n\n' for e in events).encode();self.send_response(200);self.send_header('Content-Type','text/event-stream');self.send_header('Content-Length',str(len(data)))
      if a.header!='absent':self.send_header('X-Reasoning-Included',a.header)
      self.end_headers();self.wfile.write(data)
    mock=http.server.ThreadingHTTPServer(('127.0.0.1',0),Mock);threading.Thread(target=mock.serve_forever,daemon=True).start()
    (home/'config.toml').write_text('model="gpt-5"\nmodel_provider="mock"\nweb_search="disabled"\nmodel_context_window=100000\nmodel_auto_compact_token_limit='+str(a.limit)+'\nmodel_post_turn_compact_threshold_percent=0\n[features]\nplugins=false\n[model_providers.mock]\nname="Local mock"\nbase_url="http://127.0.0.1:'+str(mock.server_port)+'/v1"\nwire_api="responses"\nrequires_openai_auth=false\nrequest_max_retries=0\nstream_max_retries=0\n')
    env={'PATH':os.environ['PATH'],'HOME':str(root),'CODEX_HOME':str(home),'TMPDIR':str(root),'RUST_LOG':'warn'};err=(root/'stderr.log').open('w');proc=subprocess.Popen([a.binary,'app-server','--stdio'],cwd=cwd,env=env,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=err,text=True,bufsize=1);q=queue.Queue();seq=0
    def reader():
     for line in proc.stdout:
      try:q.put(json.loads(line))
      except ValueError:pass
    threading.Thread(target=reader,daemon=True).start()
    def send(v):proc.stdin.write(json.dumps(v)+'\n');proc.stdin.flush()
    def call(m,p):
     nonlocal seq
     seq+=1;send({'id':seq,'method':m,'params':p});deadline=time.monotonic()+25
     while time.monotonic()<deadline:
      v=q.get(timeout=max(.1,deadline-time.monotonic()))
      if v.get('id')==seq:
       if 'error' in v:raise RuntimeError(v['error'])
       return v['result']
      notices.append(v)
     raise TimeoutError(m)
    try:
     call('initialize',{'clientInfo':{'name':'compaction_probe','version':'1'},'capabilities':{'experimentalApi':True}});send({'method':'initialized','params':{}});tid=call('thread/start',{'cwd':str(cwd),'sandbox':'danger-full-access','approvalPolicy':'never'})['thread']['id'];turns=[]
     for i in range(3):
      call('turn/start',{'threadId':tid,'input':[{'type':'text','text':'Reply '+str(i)+'.'}]});deadline=time.monotonic()+30
      while time.monotonic()<deadline:
       v=q.get(timeout=max(.1,deadline-time.monotonic()));notices.append(v)
       if v.get('method')=='turn/completed':turns.append({'status':v['params']['turn']['status'],'error':v['params']['turn'].get('error')});break
      else:raise TimeoutError('turn complete')
     proc.terminate();proc.wait(timeout=5)
     shapes=[]
     for b in requests:
      shapes.append({'tools_count':len(b.get('tools',[])),'tool_choice':b.get('tool_choice'),'reasoning_setting':b.get('reasoning'),'input_reasoning_items':sum(x.get('type')=='reasoning' for x in b.get('input',[])),'input_reasoning_bytes':sum(len(x.get('encrypted_content','')) for x in b.get('input',[]) if x.get('type')=='reasoning'),'server_reported_total':None if a.strict_empty_tools and not b.get('tools') else 110})
     compactions=0;usage=[]
     for p in (home/'sessions').rglob('rollout-*.jsonl'):
      for line in p.read_text().splitlines():
       r=json.loads(line);pay=r.get('payload',{})
       if r.get('type')=='compacted':compactions+=1
       if pay.get('type')=='token_count':usage.append(pay.get('info'))
     (root/'requests.json').write_text(json.dumps(requests));(root/'notifications.json').write_text(json.dumps(notices));print(json.dumps({'private_root':str(root),'binary':a.binary,'reasoning_fixture':a.reasoning,'reasoning_header':a.header,'strict_empty_tools':a.strict_empty_tools,'auto_compact_limit':a.limit,'turns':turns,'requests':shapes,'compactions':compactions,'token_usage':usage},indent=2))
    finally:
     proc.terminate()
     try:proc.wait(timeout=5)
     except subprocess.TimeoutExpired:proc.kill();proc.wait()
     err.close();mock.shutdown()
   if __name__=='__main__':main()