Idea: add a deterministic `type: mcp` step for direct MCP tool calls

[hertznsk]

Summary

Add a first-class deterministic workflow step that calls one tool on a configured MCP server directly, without asking an LLM agent to choose or invoke the tool.

This would let workflows use the existing MCP integration ecosystem for operations such as querying issue trackers, updating repository metadata, reading databases, or calling internal services while keeping the operation visible and reviewable in YAML.

This is an idea for discussion, not a commitment to a particular final syntax.

Motivation

Conductor is strongest when probabilistic work and deterministic orchestration are kept separate. Today, deterministic integration work generally falls back to type: script:

- name: update_tracker
  type: script
  command: uv
  args: [run, python, -m, scripts.update_tracker]

That works, but the actual operation moves out of the workflow into Bash, Python, or TypeScript code. A reviewer can no longer understand the full workflow from the YAML alone, and each repository must own and maintain its integration glue.

Platforms such as n8n address this with native integrations, but building and maintaining a comparable catalog specifically for Conductor would be expensive. MCP already provides a standardized tool discovery and invocation surface across a large ecosystem of servers. A direct MCP step could therefore serve as a broad integration escape hatch while native steps remain available for common operations where Conductor can provide a better contract and user experience.

Proposed MVP shape

workflow:
  runtime:
    mcp_servers:
      tracker:
        type: stdio
        command: tracker-mcp
        env:
          TRACKER_TOKEN: ${TRACKER_TOKEN}

agents:
  - name: create_issue
    type: mcp
    server: tracker
    tool: create_issue
    arguments:
      project_id: 105
      subject: "{{ triage.output.title }}"
      description: "{{ triage.output.details }}"
    routes:
      - to: notify

The step should:

• invoke exactly one named tool without an LLM call;
• recursively render arguments against workflow context while preserving JSON-native values rather than coercing every value to a string;
• verify that server exists in runtime.mcp_servers;
• verify that the tool is permitted by the servers configured tools allowlist;
• expose the MCP result in workflow context for downstream templates and routes;
• support the normal step lifecycle in the main loop, parallel groups, and for_each where safe;
• emit dedicated started/completed/failed events for CLI, JSONL, dashboard, and replay visibility;
• apply the existing per-result MCP output limit where applicable.

A possible output envelope is:

create_issue.output:
  content: [...]       # MCP content blocks, represented as JSON-safe data
  structured: {...}    # structuredContent when supplied by the server
  is_error: false      # MCP tool execution error flag

The exact output shape needs discussion. It should preserve structured MCP data rather than flattening everything to text. Optional output: validation could apply to a documented portion of that envelope.

Why this appears feasible in the current architecture

Conductor already has most of the required building blocks:

• runtime.mcp_servers and MCPServerDef describe stdio, HTTP, and SSE servers.
• src/conductor/mcp/manager.py::MCPManager is a host-side MCP client that can connect, discover tools, call tools, manage process lifetime, and apply tool-output limits.
• WorkflowEngine already dispatches non-LLM step types such as script, set, and wait and stores their typed outputs in WorkflowContext.
• Existing routing, limits, checkpointing, event emission, and output validation can be reused around a new executor.

The direct step should be owned by the workflow engine/executor layer, not by an LLM provider. Current providers expose MCP through different mechanisms: Claude uses the host-side MCPManager, while Copilot and Claude Agent SDK delegate MCP configuration to their SDK/CLI. A deterministic step must have one provider-independent execution path and must also work in workflows whose LLM provider does not support MCP tools.

Important corrections and constraints

No transactional guarantee

MCP standardizes tool invocation, not the semantics or atomicity of the external operation. Closing an MCP process or connection does not imply rollback. A successful tool may have committed an external side effect before Conductor receives, stores, or checkpoints the result.

Likewise, if a workflow is cancelled after a mutating tool call but before the step is committed to context, resume may invoke it again. The step therefore has at-least-once risk around interruption and resume unless the tool itself supports idempotency keys or reconciliation.

No implicit retries in the MVP

Conductor cannot infer whether an arbitrary MCP tool is safe to repeat. Automatic retries could duplicate issues, comments, payments, commits, or other mutations. The first version should not retry tool calls implicitly. Any future retry support should be explicit and documented around idempotency.

Start with the transport the shared client actually supports

Although MCPServerDef accepts stdio, HTTP, and SSE, the shared host-side MCPManager currently implements stdio only. A coherent first implementation could therefore support stdio and return a clear validation error for other transports. HTTP/SSE can follow by extending the shared manager while preserving the step syntax.

Secrets and observability need an explicit policy

MCP does not redact secrets for Conductor. Tool arguments may contain credentials or sensitive payloads, and the current tool event formatting includes arguments. A direct step should not log full arguments or results by default. Events can safely include server/tool names, argument keys, elapsed time, result size, and truncation metadata; displaying values should require an explicit, redaction-aware policy.

Concurrency needs a defined ownership model

A workflow-owned connection pool could keep one stdio server alive and reuse it across direct calls. Calls sharing the same server/session should initially be serialized unless client/server concurrency is known to be safe; different servers or working directories can proceed independently. Cleanup must be bounded by workflow lifecycle and robust under cancellation.

Why sequence is not part of the MVP

A special multi-call sequence syntax is tempting for stateful servers, but it would immediately add nested scoping, conditional execution, partial failure policy, cleanup hooks, and misleading expectations of transactionality.

Ordinary workflow steps already express ordered calls and routing. Connection reuse can be an executor lifecycle detail, so consecutive type: mcp steps may share the same live stdio server without introducing another programming language inside YAML.

If real servers demonstrate a need for an indivisible connection-scoped block, sequence syntax can be discussed later as non-transactional convenience sugar.

Relationship to #222 (type: http)

This proposal complements rather than replaces #222.

A native HTTP step remains preferable for direct REST calls because Conductor can provide HTTP-specific behavior that a generic MCP tool cannot guarantee consistently:

• typed request and response bodies;
• status codes, headers, and non-2xx payloads as a stable output contract;
• method-aware retry defaults;
• standard authorization-header redaction;
• lower startup and serialization overhead;
• no dependency on the quality, lifecycle, or security behavior of a third-party MCP HTTP client server.

The MCP step is the broad integration escape hatch for existing MCP tools. The HTTP step is the purpose-built primitive for a very common protocol.

Suggested phased scope

Phase 1

• one type: mcp step equals one tool call;
• stdio servers through an engine-owned executor/client lifecycle;
• typed recursive argument rendering;
• structured result preservation and optional output validation;
• no implicit retry and no sequence syntax;
• safe events, cancellation cleanup, routing, parallel/for-each support with per-session serialization;
• validation that the server exists and the tool is allowed.

Phase 2

• host-side Streamable HTTP/SSE transport support;
• transport-specific authentication and lifecycle handling;
• richer static validation using discovered tool schemas where practical.

Later, only if justified by real workflows

• explicit idempotency-aware retry policy;
• connection-scoped sequence convenience syntax;
• declarative redaction annotations for selected argument/result paths.

Open questions

• Should the step type be named mcp, mcp_call, or mcp_tool?
• What stable JSON-safe representation should be used for MCP content blocks and structuredContent?
• Should isError: true become a failed workflow step by default, or remain routable output with an opt-in fail mode?
• Should a direct step reuse a long-lived workflow-level server process, or should lifecycle be scoped more narrowly by server and working directory?
• How should direct steps and provider-managed MCP servers avoid duplicate stdio processes when both use the same configuration?
• Is the Python MCP SDK promoted to a required dependency, or should this step live behind an installation extra?
• Which event fields can be shown safely without exposing tool arguments or results?

Acceptance criteria for an eventual MVP

• A workflow can call a configured stdio MCP tool without invoking an LLM provider.
• Arguments retain JSON-native types after template rendering.
• Structured MCP results remain structured and are available to routes and downstream steps.
• Unknown servers, disallowed tools, unsupported transports, and invalid step-only field combinations fail clearly.
• No automatic retries occur for tool calls.
• Cancellation closes owned resources best-effort and documents that external side effects may remain.
• Same-session calls are concurrency-safe, with serialization as the conservative default.
• Events and logs do not expose full argument/result values by default.
• The step works independently of the selected LLM provider.
• An example demonstrates an MCP call followed by normal conditional routing.

Related: #222.