Skip to content

Ultra multi-agent / Responses multi_agent does not work with regular API-key model providers #37858

Open
Open
Ultra multi-agent / Responses multi_agent does not work with regular API-key model providers#37858
Labels
CLIIssues related to the Codex CLIbugSomething isn't workingcustom-modelIssues related to custom model providers (including local models)subagentIssues involving subagents or multi-agent features

Description

@RaviTharuma
RaviTharuma
opened on Aug 10, 2026

What version of Codex CLI is running?

0.147.0 (also reproduces on recent Desktop / app-server builds)

What subscription do you have?

ChatGPT Pro (and separately OpenAI API key / custom model_providers for inference)

Which model were you using?

gpt-5.6-sol / gpt-5.6-terra / gpt-5.6-luna (and equivalent IDs via a custom Responses-compatible provider)

What platform is your computer?

macOS (also relevant to any platform using custom model_providers)

What issue are you seeing?

Full “Ultra” multi-agent / subagent orchestration does not work when inference goes through a regular API provider (custom model_providers.* with wire_api = "responses" + env_key API-key auth).

This is distinct from:

  • setting model_reasoning_effort = "ultra" (reasoning effort alone can work on API providers), and
  • ChatGPT-signed-in / first-party OpenAI provider sessions, where native Ultra multi-agent coordination works.

Observed failure modes with regular API providers:

  1. Responses API multi_agent is rejected or non-functional
    When a Responses request includes the multi-agent surface (e.g. multi_agent.enabled = true / multi-agent beta), OpenAI-compatible or non–ChatGPT-backend endpoints commonly respond with:

    • 400 / Unsupported parameter: multi_agent
    • or auth / routing errors that never produce a multi-agent tree

  2. Native Codex Ultra subagent product path is ChatGPT-backend-centric
    With a custom API-key model_provider, Ultra-level parallel OpenAI multi-agent coordination (server-side subagent tree, multi_agent_v2 product behavior tied to first-party models) does not behave like a ChatGPT-auth session. Client-side spawn_agent / local multi-agent tooling may partially work, but that is not the same as first-party Ultra multi-agent.

  3. Related hardcoded ChatGPT-backend assumptions
    GPT-5.6 Sol/Terra/Luna paths that assume ChatGPT-backend fields (e.g. responses-lite / multi_agent version headers) already cause 400s on non–ChatGPT backends — see related issues below. That further blocks “just use Ultra models + multi-agent on any Responses-compatible base_url”.

Net: users who correctly configure a regular API provider for Sol/Terra/Luna get coding + reasoning effort, but not the full Ultra multi-agent / subagent product experience.

What steps can reproduce the bug?

  1. Configure a working API-key custom provider (not ChatGPT OAuth), for example:
[model_providers.my_api]
name = "OpenAI-compatible Responses API"
base_url = "https://api.example.com/v1"
wire_api = "responses"
env_key = "MY_API_KEY"
requires_openai_auth = false

model_provider = "my_api"
model = "gpt-5.6-sol"
model_reasoning_effort = "ultra"

[features]
multi_agent_v2 = true
  1. Confirm plain inference works:
codex exec --skip-git-repo-check "Reply with exactly: API_OK"

  1. Attempt Ultra multi-agent / subagent work, e.g.:

    • ask the agent to spawn multiple specialized subagents in parallel and synthesize, or
    • send a Responses request through the same provider with multi-agent enabled.

  2. Compare with the same prompt on the default ChatGPT-authenticated OpenAI provider / profile.

What is the expected behavior?

Either:

A (preferred): Full Ultra multi-agent / native subagent orchestration works for regular API-key Responses providers when the upstream model supports it — including transparent multi-agent parameters and no ChatGPT-only internal headers required for Sol/Terra/Luna, or

B: Codex clearly documents and surfaces in product UI/CLI that Ultra multi-agent / server multi-agent is ChatGPT-backend-only, while API-key custom providers only support single-agent inference (and optionally client-local spawn), so users are not left debugging silent no-spawn / Unsupported parameter: multi_agent failures.

Today neither A nor B is complete: config accepts Ultra + multi_agent_v2 + custom providers, but the Ultra multi-agent experience is incomplete or broken on regular API providers.

Additional information

Works on API providers (in our testing):

  • Single-agent Responses inference
  • Reasoning effort including ultra (as an effort level)
  • Many built-in tools that the upstream Responses API actually implements (e.g. web_search when the endpoint supports it)

Does not work / incomplete on API providers:

  • Responses multi_agent / server-side multi-agent tree
  • Full native Ultra multi-agent product behavior comparable to ChatGPT-auth sessions
  • Paths that inject ChatGPT-backend-only multi_agent / responses-lite assumptions against a plain API provider

Related issues:

Please treat this as a product + protocol gap for regular API providers, not an Azure-only issue: any model_providers.* entry that is “just” Responses + API key hits the same Ultra multi-agent wall.

Happy to provide redacted request/response pairs (no secrets) if useful.

Activity

  1. github-actions
    added
    bugSomething isn't working
    CLIIssues related to the Codex CLI
    custom-modelIssues related to custom model providers (including local models)
    subagentIssues involving subagents or multi-agent features
    on Aug 10, 2026

  2. github-actions commented on Aug 10, 2026

    @github-actions
    github-actionsbot
    on Aug 10, 2026 – with GitHub Actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  3. CCanxue commented on Aug 11, 2026

    @CCanxue
    CCanxue
    on Aug 11, 2026

    Data point for the client-side part: non-OpenAI parent -> non-OpenAI child V2 spawn / followup works reliably once the task is delivered as a plain user message (V1-style UserInput); we verified this with deepseek-v4-flash. OpenAI parent -> third-party child remains broken because the assignment is real ciphertext (#36376 / #37197) and needs a parent-side plaintext delivery path. Patch and evidence: https://github.com/CCanxue/codex-deepseek-subagent-fix

  4. rnett commented on Aug 11, 2026

    @rnett
    rnett
    on Aug 11, 2026

    Additional data point aligning with this issue for custom API-key providers (Windows, Codex Desktop).

    Versions

    • codex-cli 0.147.0, Desktop MSIX OpenAI.Codex_26.803.10989.0_x64 (internal 26.803.81509), custom model_provider = "litellm", auth mode ApiKey (no ChatGPT/OpenID token).

    Headless path works; the Desktop app UI does not

    • Headless sub-agent spawning works end-to-end: codex exec on the same custom provider works, and the CLI/rollout layer shows sub-agent activity (rollout source breakdown includes subagent:thread_spawn, subagent:other, subagent:review entries).
    • For this same setup, the Desktop app UI reports sub-agents as not working.
    • Workaround in use on our side: a proxy namespace_flatten_callback that rewrites/routes the requests through the custom provider so plain inference and headless spawning succeed.

    Observed app behavior

    • The custom-only ApiKey provider is configured and plain inference works (both headless and, partially, in the app).
    • Sub-agent orchestration via the app UI does not succeed, while the same custom provider path works headless.
    • Related: the app-level collaboration.spawn_agent / MultiAgentV2 tool_namespace interplay is exactly the edge the flatten callback has to compensate for here.

    Happy to provide redacted request/response pairs if useful. No credentials included.

  5. jdcodes1 commented on Aug 18, 2026

    @jdcodes1
    jdcodes1
    on Aug 18, 2026

    Adding the client-side mechanism that makes API-key multi-agent structurally broken today, beyond server-side gating: inter-agent task payloads are shipped as encrypted_content blobs that only the originating ChatGPT-backed session context can decrypt server-side. The client never decrypts them — InterAgentCommunication::to_model_input_item emits a plaintext header plus an EncryptedContent item and trusts the backend to expand it (protocol/src/protocol.rs#L813-L846). With a custom model_providers.* + env_key setup there is no such backend context, so even when spawn mechanics succeed, the sub-agent receives an empty payload and idles — the exact silent failure documented in #37822, and the cross-provider variant in #38656.

    So making Ultra multi-agent work on API providers needs two things, not one: (a) whatever server-side multi_agent capability gating applies, and (b) a plaintext inter-agent message path when the participants aren't on the encrypting backend — the constructor for it already exists (InterAgentCommunication::new vs new_encrypted). Until then, the documentation/gating ask in #37859 seems right: fail at spawn time with a clear "multi-agent requires ChatGPT auth / first-party provider" instead of half-working.

  6. RaviTharuma commented on Sep 3, 2026

    @RaviTharuma
    RaviTharuma
    on Sep 3, 2026
    Author

    Traced the encryption gating you and CCanxue both pointed at, down to the actual call site — this closes the loop between "ciphertext" (CCanxue) and "no decrypting backend" (jdcodes1):

    codex-rs/core/src/tools/handlers/multi_agents_v2.rs::communication_from_tool_message picks encrypted vs. plaintext based on ToolCallSource:

    if !matches!(source, crate::tools::context::ToolCallSource::DirectPlaintextMessage) {
    return InterAgentCommunication::new_encrypted(author, recipient, Vec::new(), message, trigger_turn);
}

    And ToolCallSource::DirectPlaintextMessage is set in codex-rs/core/src/tools/router.rs::ToolCall::direct_source, gated on whether encrypted_function_args for that spawn_agent/send_message/followup_task call came back empty:

    if self.tool_name.namespace.as_deref() == Some("collaboration")
    && matches!(self.tool_name.name.as_str(), "spawn_agent" | "send_message" | "followup_task")
    && self.encrypted_function_args.as_ref().is_some_and(Vec::is_empty)
{
    ToolCallSource::DirectPlaintextMessage
} else {
    ToolCallSource::Direct
}

    So the plaintext path isn't dead code gated on provider/auth type at all — it's driven by whether the upstream response returned encrypted_function_args for that tool call. On a first-party ChatGPT-auth backend the Responses API returns those args encrypted (so Direct, so new_encrypted, so the payload rides as AgentMessageInputContent::EncryptedContent that only that backend can expand — matches your and rnett's observed idle/empty-payload symptom exactly). Against a custom model_providers.* + env_key backend, whether encrypted_function_args comes back empty depends entirely on what that backend's Responses-compatible endpoint chooses to populate — nothing in the client forces it either way, which is consistent with CCanxue's deepseek-v4-flash data point working (their backend evidently returns empty encrypted_function_args, so the client takes the plaintext branch) while your litellm/custom-provider setup didn't.

    Concretely this means: (1) InterAgentCommunication::new(...) (fully plaintext, no backend decrypt needed) is real, reachable code, not aspirational — CCanxue's fix report of it working end-to-end for a non-OpenAI backend is consistent with the source; (2) whether your provider hits it is a function of what your Responses-compatible server puts in encrypted_function_args for collaboration.* tool calls, not something you can force from model_providers config today. What's still needed from a maintainer: either document that self-hosted Responses-compatible servers must return empty encrypted_function_args for collaboration tool calls to get plaintext delivery, or add an explicit provider-capability flag so this doesn't depend on emergent backend behavior. Server-side multi_agent capability gating (the other half you and jdcodes1 flagged) is a separate, unverifiable-from-client-source question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingcustom-modelIssues related to custom model providers (including local models)subagentIssues involving subagents or multi-agent features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions