Skip to content

Non-vision models blocked from passing images to vision-capable MCP tools #29216

Closed as not planned
Closed as not planned
Non-vision models blocked from passing images to vision-capable MCP tools#29216
Assignees
kitlangton

Description

@andrea-tomassi
andrea-tomassi
opened on May 25, 2026

Description

When using a non-vision model, image attachments are rejected at the backend level before the LLM can process them. The model receives an error text instead of the image, so it can't delegate the image to a vision-capable MCP tool (like zai-mcp-server_analyze_image).

This is inconsistent: a vision model and a non-vision model are both capable of calling the same MCP tool to analyze an image — neither uses its own "vision" for the actual analysis. But only the vision model can receive the image reference in the first place.

Steps to Reproduce

  1. Select a non-vision model (e.g., one without image in its modalities.input)
  2. Upload an image via drag-and-drop or the attach button (this works — frontend gating doesn't exist)
  3. Submit the prompt
  4. The LLM receives: ERROR: Cannot read "image.png" (this model does not support image input). Inform the user.
  5. The LLM cannot call zai-mcp-server_analyze_image (or any other vision tool) because the image reference is gone

Root Cause

In packages/opencode/src/provider/transform.ts, the unsupportedParts() function (lines 391–427) replaces image/file parts with an error text when the model lacks capabilities.input.image:

// line 414-416
const modality = mimeToModality(mime)         // → "image"
if (!modality) return part                     
if (model.capabilities.input[modality]) return part  // passes through

// line 418-421 — image stripped, replaced with error
return {
  type: "text" as const,
  text: `ERROR: Cannot read ${name} (this model does not support ${modality} input). Inform the user.`,
}

This is called at line 430 on every message before dispatch.

Meanwhile, the frontend (packages/app/src/components/prompt-input.tsx) has zero awareness of model capabilities — the attach button is always active, drag-and-drop always works, paste always works. This creates a misleading UX where you can upload but the backend silently strips it.

Expected Behavior

A non-vision model should still receive the image as context (even if only as a text reference to the file), so it can dispatch the image to a vision-capable MCP tool (e.g., zai-mcp-server_analyze_image). The model itself doesn't need to support image input — it just needs to know the image exists and have a reference to pass to a tool.

Suggested Fix

Instead of replacing unsupported image parts with a hard error, pass the file metadata as a text part (e.g., [Attached image: "image.png" (image/png)]) so the LLM can still reason about it and route it to tools that handle images.

Environment

  • OpenCode version: current dev branch
  • Reproducible: always

Activity

  1. github-actions
    assigned
    kitlangton
    on May 25, 2026

  2. github-actions commented on May 25, 2026

    @github-actions
    github-actionsBot
    on May 25, 2026 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

    If your issue is distinct, please clarify what differentiates it from the above. Otherwise, consider adding a comment to the most relevant existing issue instead.

  3. andrea-tomassi commented on May 25, 2026

    @andrea-tomassi
    andrea-tomassi
    on May 25, 2026
    Author

    Thanks for flagging these. #29216 is distinct from each:

    vs #24948 and #22828 (auto vision fallback / auto transcription)

    Both propose a server-side automation — automatically call a separate vision model to transcribe the image into text, then feed that text to the non-vision model. This is a heavyweight feature with config, model selection, and a processing pipeline.

    #29216 is much simpler and more fundamental: just don't strip the image. The non-vision model doesn't need to read the image itself — it only needs the image reference (file path / MIME / filename) so it can decide to dispatch the image to a vision-capable MCP tool. This is the tool-delegation pattern — the model says "I can't see this, let me call analyze_image on it." An auto-fallback does the work silently; the tool-delegation pattern gives the model agency.

    vs #23508 (provide full image path in error)

    The closest in spirit, but #23508 asks for a better error message. #29216 argues there shouldn't be an error at all. The image should pass through as context the model can reason about, not as an error it must apologize for. The difference: #23508 says "tell me where the image is" → #29216 says "don't destroy the image reference in the first place."

    vs #26668 (clipboard auto-read)

    Unrelated bug — that's about clipboard images being auto-attached even for text-only messages. #29216 is about intentionally uploaded images being stripped.

    Summary: All four existing issues assume the model must either natively support images or get a text transcription. They all work within the assumption that the unsupportedParts() rejection is correct behavior. #29216 challenges that assumption: the model doesn't need to support images — it just needs an un-destroyed reference so it can route to tools.

    The fix is also simpler than #24948/#22828: change unsupportedParts() to emit a text part with file metadata instead of a hard error — no model routing, no config, no transcription pipeline needed.

  4. wha7ev9r
    marked [FEATURE]:Expose image file path alongside [image] placeholder so non-vision models can use MCP vision tools #31221 as a duplicate of this issue on Jun 7, 2026

  5. valorisa commented on Jul 3, 2026

    @valorisa
    valorisa
    on Jul 3, 2026

    +1, hitting the exact symptom described here.

    Setup: non-vision model as primary (text-only, cost-optimized), with a vision-capable MCP tool configured and available. Pasting a screenshot triggers the hard error from unsupportedParts() before the model ever gets a chance to route it to the MCP tool — so the vision MCP tool sits there configured and unused, and the image is just dropped with an error instead.

    This is the same root cause I flagged on #19853: the image never reaches agent-level routing logic, because it's intercepted and rejected at the transform layer first.

    Real-world impact on my workflow: I frequently need to share terminal error screenshots, UI/UX bugs from the browser, and architecture diagrams while using a non-vision model as my daily driver. Right now the only workarounds are describing the screenshot in text (lossy, often misses the actual issue) or uploading to a third-party service and sharing a link (extra friction, breaks the flow). Both defeat the point of having a vision-capable MCP tool configured in the first place.

    The minimal fix proposed here (pass a text reference to the image instead of hard-erroring) seems like the right scope — it doesn't touch vision-model behavior at all, it just stops swallowing the image before non-vision-model agent code can decide what to do with it.

  6. richard-fairthorne commented on Jul 7, 2026

    @richard-fairthorne
    richard-fairthorne
    on Jul 7, 2026

    +1 — do less.

  7. 382960875 commented on Jul 22, 2026

    @382960875
    382960875
    on Jul 22, 2026

    +1 on this — same issue with the oh-my-openagent plugin scenario, where the fix is even more achievable because the vision-capable agent already exists.

    Setup:

    • Model: deepseek-v4-pro (text-only, capabilities.input.image = false)
    • Plugin: oh-my-openagent@latest — provides multimodal-looker agent + look_at tool with vision fallback chain (gpt-5.5 → kimi-k2.6 → glm-4.6v)
    • Action: paste screenshot (Ctrl+V) into chat

    What happens:

    1. TUI saves clipboard to image.png, read tool returns it as a file attachment
    2. unsupportedParts() strips the image and replaces with "ERROR: Cannot read image.png (this model does not support image input)"
    3. The model sees only the error text — no file metadata, no path, nothing to delegate
    4. multimodal-looker agent and look_at tool are available but unreachable because the image reference was destroyed

    Why this issue's proposed fix helps:
    If unsupportedParts() emitted file metadata ([Attached image: "image.png" (image/png)]) instead of a hard error, the model could then delegate to multimodal-looker or call look_at — the vision pipeline already exists in oh-my-openagent, it's just blocked by the stripped reference.

    The broader picture:
    There are 7+ issues requesting auto-vision-fallback (#24948, #22828, #21004, #18437, #31936, #26160). Most were closed as not_planned or auto-closed. This issue's approach — don't destroy the reference — is the minimal, highest-impact fix that unblocks all of them without requiring a full auto-routing pipeline. A model that can see [image.png attached] can choose to route it to any available vision tool, MCP or oh-my-openagent's built-in multimodal-looker.

  8. ndj888 commented on Jul 22, 2026

    @ndj888
    ndj888
    on Jul 22, 2026

    Additional reproducible evidence from OpenCode TUI v1.18.2 on macOS arm64:

    Reproduction

    1. Copy an image to the clipboard and paste it into the TUI.
    2. The prompt shows [Image 1].
    3. Ask an agent to pass that attachment to a local image tool whose API requires a readable filesystem path, such as an img2img script.
    4. The model may receive the pixels when vision input is enabled, but it has no usable path to pass to the tool. With a text-only model, unsupportedParts() removes the image before tool delegation as already described in this issue.

    Evidence

    For clipboard images, packages/tui/src/component/prompt/index.tsx constructs a FilePart with:

    • filename: "clipboard"
    • url: "data:image/...;base64,..."
    • source.path: "clipboard" because no filepath exists

    On macOS, packages/tui/src/clipboard.ts briefly writes $TMPDIR/opencode-clipboard.png, reads it as base64, and deletes it in finally. That temporary path is therefore not a stable, tool-addressable reference and is not propagated with the attachment.

    Local session records confirm pasted images are persisted as image data URLs without a usable absolute path. This is still reproducible in v1.18.2.

    Impact beyond direct vision

    This also blocks local path-based tools even when the active model can see the image. Seeing the pixels and being able to invoke tool --input-image /absolute/path.png are separate capabilities. [Image 1], clipboard, and a data URL cannot be passed to tools that accept only local paths.

    A privacy-safe fix could preserve file metadata for non-vision models and offer an explicit attachment materialization mechanism for local tools:

    • If a trustworthy source path exists, expose it only through an explicit user action.
    • If the clipboard has pixels but no source path, materialize them into an app-managed temporary file with a defined lifetime.
    • Do not automatically inject absolute paths into remote model context, since they can disclose usernames and filesystem structure.
    • Do not assume a local path is usable by a remote MCP server.

    This overlaps #31221, which was closed as a duplicate of this issue, but adds a verified current-version reproduction and the local img2img/tool handoff case.

  9. github-actions commented on Sep 22, 2026

    @github-actions
    github-actionsBot
    on Sep 22, 2026 – with GitHub Actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

  10. github-actions
    closed this as not plannedon Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions