Codex CLI 0.147.0 rejects Amazon Bedrock input and corrupts subagent handoff #37475

Activity

1. github-actions commented on Aug 7, 2026

   

   github-actionsBot

   on Aug 7, 2026 – with GitHub Actions

   Contributor

   More actions

   Potential duplicates detected. Please review them and close your issue if it is a duplicate.

   • Subagent task payload invisible to custom non-OpenAI providers (DeepSeek): encrypted_content block dropped with multi_agent_version v2 #36586
   • [Bug] spawn_agent/followup_task/send_message: child agents receive empty task payload (multi-agent v2, 0.146.0-alpha.9.2) #36321
   • Sub-agent task messages are delivered with an empty payload (Codex Desktop) #36493

   Powered by Codex Action
2. github-actions

   added

   bugSomething isn't workingSomething isn't working

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   aws-bedrockIssues related to AWS Bedrock providerIssues related to AWS Bedrock provider

   subagentIssues involving subagents or multi-agent featuresIssues involving subagents or multi-agent features

   on Aug 7, 2026
3. xiehust commented on Aug 8, 2026

   

   xiehust

   on Aug 8, 2026

   More actions

   Additional confirmed reproduction with source-level evidence.

   Environment:

   • codex-cli 0.147.0
   • Linux Ubuntu 24.04, aarch64 (6.17.0-1019-aws)
   • built-in amazon-bedrock provider
   • model openai.gpt-5.6-sol
   • MultiAgentV2
   • source checkout 3aae5d885bac39c1262491aa3fd100dfd8b3919f

   I reproduced the same validation error from a real interactive rollout. The failure is specifically tied to the internal agent_message input variant.

   The parent successfully received a namespaced collaboration call:

   {
     "type": "function_call",
     "name": "spawn_agent",
     "namespace": "collaboration",
     "arguments": "{...nonempty message...}",
     "call_id": "call_..."
   }

   The spawned child then persisted this as its initial model-visible input:

   {
     "type": "agent_message",
     "author": "/root",
     "recipient": "/root/issue15_final_check",
     "content": [
       {
         "type": "input_text",
         "text": "Message Type: NEW_TASK\nTask name: /root/issue15_final_check\nSender: /root\nPayload:\n"
       },
       {
         "type": "encrypted_content",
         "encrypted_content": "<the plaintext delegated task>"
       }
     ]
   }

   Before any child model output, Bedrock rejected the request:

   validation_error: invalid request body: Invalid input: value did not match any expected variant
   

   There is a second, more damaging failure mode: Codex reports the child error back to the parent as another type: "agent_message". That item is retained in parent history. The next ordinary user message in the parent thread then fails immediately with the same validation error, so the parent thread becomes effectively unresumable.

   Relevant current code paths:

   • codex-rs/protocol/src/protocol.rs: InterAgentCommunication::to_model_input_item emits ResponseItem::AgentMessage.
   • codex-rs/core/src/tools/router.rs: ToolCall::direct_source selects plaintext collaboration only when encrypted_function_args == Some([]).
   • Bedrock returns the task text without that marker, so it is classified as encrypted even though the value is plaintext.
   • codex-rs/core/src/client.rs: the non-OpenAI request cleanup clears metadata and encrypted_function_args, but does not translate or remove ResponseItem::AgentMessage before serializing the Responses request.

   This appears to require both provider-aware collaboration message classification and a non-OpenAI request-boundary representation for agent_message. Regression coverage should include:

   1. Bedrock parent spawning a child with a nonempty task.
   2. The child initial request containing only Bedrock-supported input variants.
   3. Child completion/error delivery back to the parent.
   4. A subsequent normal parent turn remaining usable.

   The duplicate Started lines shown by the TUI are incidental; the provider rejection is reproducible from the persisted request history itself.
4. CCanxue commented on Aug 11, 2026

   

   CCanxue

   on Aug 11, 2026

   More actions

   Additional data point: the provider-incompatible agent_message item also causes strict validation failures on non-ChatGPT backends (Bedrock rejects it with Invalid 'input' on 0.147.0), not just silent payload drops on DeepSeek. This matches the pattern that agent_message / encrypted_content are Codex-specific protocol assumptions.

   For non-OpenAI providers we deliver V2 spawn / followup_task / send_message payloads as plain user messages instead, which avoids the agent_message item entirely and is verified end-to-end: https://github.com/CCanxue/codex-deepseek-subagent-fix
5. blueCycle commented on Aug 19, 2026

   

   blueCycle

   on Aug 19, 2026

   More actions

   Can confirm this also reproduces on 0.147.0 via a different trigger path than the original repro: instead of a manual spawn_agent prompt, this happened when an autonomous multi-step task organically spawned a sub-agent mid-session (via InterAgentCommunication) on model openai.gpt-5.6-sol. Every subsequent turn on that same thread then failed identically — the malformed subAgentActivity/agent_message item stays in conversation history and gets re-sent on every turn, so the thread becomes permanently stuck (not just the single spawn call).

   Client-side trace showed the failing POST to bedrock-mantle.us-east-1.api.aws/openai/v1/responses carrying a ~530KB input array; server returned:

   {"error":{"code":"validation_error","message":"invalid request body: Invalid 'input': value did not match any expected variant","param":null,"type":"invalid_request_error"}}
   

   This lines up with #33551's diagnosis — sol/terra force Multi-Agent V2 regardless of the multi_agent_v2 config flag, and V2's agent_message/subAgentActivity item types aren't recognized by Bedrock Mantle's Responses schema. Downgrading to 0.146.1 avoids it entirely (subagent spawn works normally there).

   Would be good to see this and #33551 linked/triaged together since they're the same root cause with two different repro paths.
6. lazydragon commented on Aug 19, 2026

   

   lazydragon

   on Aug 19, 2026

   More actions

   This workaround works for me. Set model catalog and set multi_agent_version to v1

   {
   "models": [
   {
   "slug": "openai.gpt-5.6-terra",
   "display_name": "GPT-5.6 Terra",
   "description": "GPT-5.6 Terra on Bedrock Mantle.",
   "priority": 1,
   "visibility": "list",
   "supported_in_api": true,
   "shell_type": "default",
   "base_instructions": "",
   "supported_reasoning_levels": [],
   "supports_reasoning_summaries": true,
   "support_verbosity": true,
   "supports_parallel_tool_calls": true,
   "experimental_supported_tools": [],
   "truncation_policy": { "mode": "tokens", "limit": 272000 },
   "context_window": 272000,
   "web_search_tool_type": "text",
   "multi_agent_version": "v1"
   }
   ]
   }

   Point model_catalog_json at it and set model to the slug:

   model = "openai.gpt-5.6-terra"
   model_provider = "amazon-bedrock"
   model_catalog_json = "/absolute/path/to/catalog.json"

   👍React with 👍3Reacted by Andrew (Qingjie) Ren, Dmitry Shumilin and William
7. tbillow-pnnl commented on Aug 21, 2026

   

   tbillow-pnnl

   on Aug 21, 2026

   Last edited by tbillow-pnnl

   More actions

   Additional independent reproduction on a newer Codex Desktop/CLI build.

   Environment:

   • Codex Desktop 26.810.52044 (build 6662)
   • bundled codex-cli 0.148.0-alpha.9
   • macOS 26.6.1, arm64
   • built-in amazon-bedrock provider
   • model openai.gpt-5.6-sol
   • Bedrock Mantle region us-east-1

   I reproduced this repeatedly across separate tasks on August 19-21, 2026. In each case, the parent successfully invoked spawn_agent, but the child rollout started with an internal agent_message whose content contained:

   input_text: Message Type: NEW_TASK ... Payload:
   encrypted_content
   

   The child failed before producing any model output:

   validation_error: invalid request body: Invalid 'input': value did not match any expected variant
   

   Codex then delivered that failure back to the parent as another agent_message. In the Desktop UI, the turn became a system error and the user saw repeated validation-error toasts, effectively killing the active workflow.

   Workaround validation update

   Disabling multi-agent is a safe containment workaround:

   [features]
   multi_agent = false

   I also tested the model-catalog workaround described in this thread: force the Bedrock model to use multi_agent_version: "v1" and keep multi-agent enabled.

   {
     "models": [
       {
         "slug": "openai.gpt-5.6-sol",
         "multi_agent_version": "v1"
       }
     ]
   }

   Using a temporary catalog derived from Codex's current Sol model definition, I ran a real end-to-end smoke test on the same machine and Bedrock configuration:

   1. Started a fresh task with multi-agent enabled and Sol forced to v1.
   2. The parent spawned exactly one child with a nonempty task.
   3. The parent waited for completion.
   4. The child received the task and returned OK.
   5. The parent reported success with no validation error.

   This confirms that v1 restores functional Bedrock subagents on 0.148.0-alpha.9; the issue is specific to the v2 wire representation rather than Bedrock model inference generally. The multi-agent version is fixed when a task starts, so this workaround requires a fresh task and will not repair an existing v2-poisoned thread.

   Current upstream source also contains a Bedrock catalog override that sets MultiAgentVersion::V1 because Bedrock does not support the response items used by v2 (codex-rs/model-provider/src/amazon_bedrock/catalog.rs). That suggests the v1 fallback is the intended compatibility behavior, although it is not effective in this bundled Desktop build.

   I can provide sanitized rollout excerpts if a maintainer needs additional evidence, but the full rollouts contain private project context.
8. alvaroguirs commented on Aug 22, 2026

   

   alvaroguirs

   on Aug 22, 2026

   More actions

   I can reproduce this issue in the Codex macOS app when using Amazon Bedrock.

   Environment

   • Codex App: 26.818.21641
   • Bundled runtime/CLI shown by diagnostics: 0.148.0-alpha.21
   • Provider: amazon-bedrock
   • Model: openai.gpt-5.6-sol

   Reproduction

   • A minimal spawn_agent test using Bedrock + GPT-5.6-Sol fails consistently.
   • Running the equivalent subagent test through the OpenAI provider succeeds.
   • Disabling multi_agent_v2 does not resolve the issue.

   Observed behavior

   The Bedrock subagent appears to receive the delegated task as plaintext inside an encrypted_content input variant. The subagent starts, but the handoff fails and it cannot process the delegated task correctly.

   This looks consistent with the issue reported here, but I wanted to confirm that it is still reproducible in Codex App 26.818.21641 / bundled runtime 0.148.0-alpha.21.

   Happy to provide the failed Bedrock session and successful OpenAI control session if that would help with debugging.
9. celia-oai

   self-assigned this

   on Aug 25, 2026
10. celia-oai commented on Aug 25, 2026

    

    celia-oai

    on Aug 25, 2026

    Contributor

    More actions

    we are working on the v2 fix with bedrock team. for now, we rolled out patch to disable: fixed by falling back to v1 in PR #39804.

    Desktop hotfix 26.818.32112 began rolling out August 21.

    For cli only alpha has it (starting with 0.150.0-alpha.2). We'll get out a main release soon with this patch.

    👍React with 👍3Reacted by Horacio Valencia, Dmitry Shumilin and ogapan
11. righteousgambit commented on Sep 1, 2026

    

    righteousgambit

    on Sep 1, 2026

    More actions

    Confirming current behavior on a build newer than any reported above, plus a root-cause trace and a design note.

    Environment

    • Codex Desktop 26.825.51511 (build 7377), bundled codex-cli 0.151.0-alpha.7.2
    • macOS Darwin 25.5.0 arm64
    • Built-in amazon-bedrock provider, openai.gpt-5.6-sol, Bedrock Mantle us-east-1

    The v1 fallback works

    With default config, subagent spawn now succeeds end-to-end. codex exec with a two-subagent prompt completed normally:

    collab: SpawnAgent ×2 → collab: Wait → collab: CloseAgent ×2
    

    Both subagents returned their assigned values, rollouts record multi_agent_version: "v1", and there were zero errors. So #39804 is effective on this build.

    Forcing v2 still reproduces — with a different error

    Setting features.multi_agent_v2.enabled = true re-breaks it deterministically. Six subagent turns failed, each with:

    {"error":{"code":"invalid_encrypted_content","message":"invalid encrypted content",
     "param":"input","type":"invalid_request_error"}}

    Note this is not the Invalid 'input': value did not match any expected variant error in the original report. Direct probes against bedrock-mantle.us-east-1.api.aws/openai/v1/responses isolate the boundary:

    Request input contains	Result
    agent_message with only input_text parts	200
    agent_message with an encrypted_content part	400 invalid_encrypted_content

    Mantle now accepts the agent_message variant itself; only the encrypted_content part is rejected. That looks like a server-side change since the earlier reports, and it narrows the remaining gap considerably.

    Root cause

    communication_from_tool_message in codex-rs/core/src/tools/handlers/multi_agents_v2.rs routes everything except ToolCallSource::DirectPlaintextMessage through InterAgentCommunication::new_encrypted. to_model_input_item in codex-rs/protocol/src/protocol.rs then serializes that as an input_text header plus an AgentMessageInputContent::EncryptedContent part.

    Worth noting: the value in that field is plaintext at this layer. A captured agent_message from a local rollout:

    author: /root  recipient: /root/subagent_a_retry
      input_text:        "Message Type: NEW_TASK\nTask name: ...\nSender: ...\nPayload:\n"
      encrypted_content: "Reply with exactly the word ALPHA and nothing else."
    

    Evidence the plaintext path is sufficient for Bedrock

    To test whether the payload shape is the only obstacle, I ran the same build with v2 forced on, behind a local proxy that rewrote each agent_message into a plain message, folding the input_text header and the encrypted_content value into one input_text part. Result: 4 rewrites, all upstream responses 200, both subagents returned correct values, no invalid_encrypted_content.

    That is only a diagnostic, not a proposed patch — but it suggests the existing plaintext representation (InterAgentCommunication::new + InterAgentMessage::render) carries enough for subagents to function on Mantle.

    Design note

    Rather than pinning Bedrock to v1 permanently, this may fit ProviderCapabilities in codex-rs/model-provider/src/provider.rs, which already models provider-owned upper bounds (namespace_tools, image_generation, web_search, external_web_access, remote_compaction). A capability for encrypted agent-message content would let v2 select the plaintext path for any provider that cannot round-trip encrypted content, instead of a Bedrock-specific version pin — which would also cover third-party Responses providers hitting the same wall.

    Happy to supply fuller logs or rerun anything against a candidate build.