Custom OpenAI-compatible providers: image file attachments do not reach vision-capable models correctly

Summary

When using a custom OpenAI-compatible provider, image attachments sent through OpenCode sessions are not reaching the model as usable vision input.

In my case:

provider: longent
model: gpt-5.4(xhigh)
transport: OpenCode desktop / session API

The exact same provider/model can successfully answer a vision request when called directly via the provider's OpenAI-compatible /chat/completions API using image_url.

But when I send an image through OpenCode as a file part (mime: image/png), the assistant responds with:

I can't read remote-test.png here because this model doesn't support image input.

So this looks like an OpenCode adapter/translation issue in the custom provider + file attachment path, not a model capability issue.

Expected behavior

If a custom OpenAI-compatible provider/model supports image input, sending an image attachment through OpenCode should be translated into a valid multimodal request and the model should be able to read the image.

Actual behavior

OpenCode session messages with file image parts end up behaving as if the model does not support image input, even though the same model works with direct API vision calls.

Repro

1. Configure a custom OpenAI-compatible provider

Example config shape:

{
  "provider": {
    "longent": {
      "name": "Longent",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "https://longent.tech/v1",
        "apiKey": "REDACTED"
      },
      "models": {
        "gpt-5.4(xhigh)": {
          "name": "GPT-5.4 XHigh",
          "options": {
            "reasoningEffort": "xhigh"
          }
        }
      }
    }
  }
}

2. Direct provider API works

A direct request to the same provider/model with image_url succeeds.

Example payload:

{
  "model": "gpt-5.4(xhigh)",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What color is this image? Reply with one word."},
        {"type": "image_url", "image_url": {"url": "https://dummyimage.com/10x10/ff0000/ffffff.png"}}
      ]
    }
  ],
  "max_tokens": 30
}

Response is correct (red / Red).

3. OpenCode session API fails for image file attachment

Create a session, then send a message with parts:

{
  "model": {"providerID": "longent", "modelID": "gpt-5.4(xhigh)"},
  "agent": "build",
  "parts": [
    {"type": "text", "text": "What color is this image? Reply with one word."},
    {
      "type": "file",
      "mime": "image/png",
      "filename": "remote-test.png",
      "url": "https://dummyimage.com/10x10/ff0000/ffffff.png"
    }
  ]
}

Returned assistant text:

I can't read `remote-test.png` here because this model doesn't support image input.

Additional observations

I also reproduced the issue from the OpenCode desktop app by attaching a local PNG file.
In the recorded session message, the user input is stored as a file part with mime: image/png.
The session itself is definitely using the custom provider/model (providerID=longent, modelID=gpt-5.4(xhigh)).
This strongly suggests the problem is in how OpenCode maps file image parts for custom OpenAI-compatible providers.

Environment

OpenCode desktop
OpenCode session version observed: 1.3.13
macOS

Hypothesis

OpenCode may be handling image attachments correctly for some built-in providers, but for @ai-sdk/openai-compatible custom providers it looks like file image parts are not being converted into the multimodal format the upstream model expects.

Possibly the provider path needs to translate image file parts into something equivalent to image_url / image content parts for OpenAI-compatible chat requests.

This issue might be a duplicate of existing issues. Please check:

[FEATURE]: Add supportsAttachments/vision capability for custom providers #8875: Same root cause — custom providers using @ai-sdk/openai-compatible have image attachments silently stripped because vision capability (SupportsAttachments) is hardcoded for built-in providers only, not inherited by custom providers
[ModelScope] Custom Provider with Qwen Model Cannot Recognize Images - "Image Input Not Supported" #15040: Same symptom — custom OpenAI-compatible provider (ModelScope/Qwen) returns "Image Input Not Supported" even though the model officially supports vision

If your issue is distinct (e.g. specifically about the file part → image_url translation path rather than capability gating), please clarify the difference.

Thanks — I checked the suggested duplicates and added one more data point from my side.

What I verified locally:

The failing session was definitely using the custom provider/model:

providerID: longent
modelID: gpt-5.4(xhigh)

The exact same provider/model works for vision when called directly through the provider's OpenAI-compatible API with image_url.

I tested against https://longent.tech/v1/chat/completions
same model
image URL input
response correctly identified the image color

The failure happens specifically when OpenCode sends the image as a file part in a session message.

message part type: file
mime: image/png
then the assistant returns: "this model doesn't support image input"

So from my testing, this is at least one of:

custom providers are not marked as vision-capable in the OpenCode pipeline
and/or file(image/*) parts are not being translated correctly for OpenAI-compatible providers

If #8875 already covers the capability-gating root cause, then this issue may be the same bug manifesting through the session file attachment path.

If helpful, I can also provide a minimal repro using:

direct provider API success with image_url
OpenCode session API failure with parts: [text, file]

Adding root-cause analysis and a workaround after digging through the opencode source.

Root cause: the `modalities` field is required but undocumented

Capability detection for image input lives in packages/opencode/src/provider/provider.ts, roughly:

capabilities: {
  input: {
    image: model.modalities?.input?.includes("image") ?? false,
    // ...
  }
}

If modalities is not defined on a custom model entry in opencode.json, capabilities.input.image defaults to false, and image parts are stripped client-side before the request is built. The model then receives text only and often replies literally "I don't support image input", which looks like a backend capability issue but is actually opencode's own gating check.

This explains why the same backend works with a direct image_url curl but fails through opencode — the file → image_url mapping logic is never reached, because the attachment is already gone by then.

Minimal repro

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "my-kimi": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "https://your-openai-compatible-gateway/v1",
        "apiKey": "sk-..."
      },
      "models": {
        "kimi-k2.5": {
          "name": "Kimi K2.5"
          // no `modalities` → image gets stripped client-side
        }
      }
    }
  }
}

Attach an image, send → assistant responds "I don't support image input."

Workaround: declare `modalities` explicitly

"kimi-k2.5": {
  "name": "Kimi K2.5",
  "modalities": {
    "input": ["text", "image"],
    "output": ["text"]
  }
}

Restart opencode (the capability cache is built at startup, not hot-reloaded). Images now flow through correctly, verified end-to-end with Kimi K2.5 and K2.6 multimodal responses.

Suggestions

Document modalities under the "Custom provider" section at opencode.ai/docs/providers. It's currently referenced nowhere in user-facing docs, even though it's the only switch that enables vision for custom providers.
Tighten the opencode.json JSON schema so provider.*.models.* describes modalities, reasoning, toolCall, temperature, etc. explicitly. Right now the inner model-entry shape is unconstrained, so IDE completion and schema validation give users zero hint that modalities exists.
Optional UX: when an image attachment is stripped because the model lacks modalities.input.image, surface a warning. That would turn a confusing symptom ("model says it doesn't support images") into an actionable one ("opencode stripped your image because modalities.input doesn't include 'image'").

Happy to open a docs PR for the first suggestion above.

Ran into the same root cause with gemma4:26b via Ollama (@ai-sdk/openai-compatible). The fix is to explicitly declare modalities in config.json for the model — this field exists in provider.ts but is completely undocumented:

{
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": {
        "gemma4:26b": {
          "modalities": {
            "input": ["text", "image"],
            "output": ["text"]
          }
        }
      }
    }
  }
}

Without this, capabilities.input.image defaults to false for all custom provider models and images are replaced with "ERROR: Cannot read image (this model does not support image input). Inform the user." before they ever reach the model.

Allowed values for modalities.input / modalities.output: "text", "audio", "image", "video", "pdf".

Also worth noting: even with this fixed, MCP tool results containing images still go through the synthetic-user-message detour (not the native tool-result path) because @ai-sdk/openai-compatible is not in the supportsMediaInToolResults allowlist in message-v2.ts. Images arrive at the model but as a separate user turn rather than inline in the tool result.

Also worth noting: even with this fixed, MCP tool results containing images still go through the synthetic-user-message detour (not the native tool-result path) because @ai-sdk/openai-compatible is not in the supportsMediaInToolResults allowlist in message-v2.ts. Images arrive at the model but as a separate user turn rather than inline in the tool result.

That's because the completions api doesnt support tools returning images

I can confirm the same behavior with LM Studio + Gemma 4 26B.

Setup:

OpenCode Desktop / OpenCode cli
Custom provider using @ai-sdk/openai-compatible
LM Studio at http://127.0.0.1:1234/v1
Model: gemma-4-26b-a4b-it-mlx
LM Studio reports the model as type: "vlm"

Direct API calls to LM Studio worked correctly with an OpenAI-compatible image_url payload, including streaming requests. The model could describe the image successfully.

However, when attaching the same image through OpenCode, OpenCode stored the image as a file part, but the request sent to LM Studio did not include an image. Instead, LM Studio received text like:

ERROR: Cannot read "image.png" (this model does not support image input). Inform the user.

Adding modalities to the custom model config fixed it:

"gemma-4-26b-a4b-it-mlx": {
  "name": "Gemma 4 26b (local)",
  "modalities": {
    "input": ["text", "image"],
    "output": ["text"]
  },
  "limit": {
    "context": 65536,
    "output": 8192
  }
}

After this, image attachments work through OpenCode.

So this seems to be a documentation/discoverability issue for custom OpenAI-compatible vision models: without modalities.input including "image", OpenCode gates the image attachment before it reaches the provider.

Looking forward to seeing the fix merged, since this small config detail makes local VLMs work nicely with OpenCode.

I ended up patching the bundled app.asar myself using qwen 3.6 to implement the changes of this pr - and it seems to work just fine (tested with koboldcpp).

If anyone else is desperate to get it working like I was, consider doing the above. Note that you need to find the current chunk file name and exact byte strings to edit it for your version of opencode. Every update the chunk hash changes or minification tweaks the output, so you will need to find it again and repatch it.

I'm experiencing the exact same issue while using LM Studio with Gemma-4 (Vision capable).

This is a critical blocker for my professional workflow. I am using the Godot MCP (GoPeak) which has the capability to capture screenshots for visual debugging, but because of this bug in OpenCode Desktop, the model remains "blind" to those captures and claims it doesn't support image input.

It's extremely frustrating because the model works perfectly in other environments, but the OpenAI-compatible adapter in OpenCode is failing to translate image attachments correctly. This makes UI/UX development and visual error tracking nearly impossible.

The solution here: #30071 👌 works fine but there are many things that should hopefully work within the Desktop version probably automatic as a default to recognize which model capabilities so we won't need to define it manually in the opencode.jsonc file.

Please prioritize a fix for the OpenAI-compatible provider's image handling.
HUGE THANKS to anyone who help in the dev of the Desktop version, keep up the good work 🙏

fix(provider): enable image support for custom OpenAI-compatible prov…

To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

I tried new /connect to connect to digitalocean but still get this Error: 400: {"message":"unsupported multimodal content: image_url","type":"invalid_request_error"} error.

it's frustrating to say the least

Hmm

Looks like bug on digitalocean side, contacting them

@hadifarnoud

@hadifarnoud talked w/ digitalocean, I assume u tried glm-5.3-flash since that's the only model I could repro w/. Currently it doesn't support images but they are working on a fix, for now ill update our model catalog to say it doesnt support images to mitigate this issue for u.

anomalyco/models.dev#10162

on opencode v2 models list refreshes every 10 min or so so itll automatically get picked up

@rekram1-node

yes @rekram1-node that is exact model I have issue with.

Yeah, can't do anything on our end aside from marking it a non vision model so that if the model reads image files it wont brick the session. Waiting on them to fix it on their end so we can restore vision capabilities in OC.

Navigation Menu

Repository navigation

Custom OpenAI-compatible providers: image file attachments do not reach vision-capable models correctly #20802

Summary

Expected behavior

Actual behavior

Repro

1. Configure a custom OpenAI-compatible provider

2. Direct provider API works

3. OpenCode session API fails for image file attachment

Additional observations

Environment

Hypothesis

Root cause: the `modalities` field is required but undocumented

Minimal repro

Workaround: declare `modalities` explicitly

Suggestions

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Custom OpenAI-compatible providers: image file attachments do not reach vision-capable models correctly #20802

Description

Summary

Expected behavior

Actual behavior

Repro

1. Configure a custom OpenAI-compatible provider

2. Direct provider API works

3. OpenCode session API fails for image file attachment

Additional observations

Environment

Hypothesis

Activity

github-actions commented on Apr 3, 2026

GravityPoet commented on Apr 3, 2026

XIYBHK commented on Apr 21, 2026

Root cause: the modalities field is required but undocumented

Minimal repro

Workaround: declare modalities explicitly

Suggestions

f-trycua commented on May 1, 2026

rekram1-node commented on May 1, 2026

12 remaining items

ngutech21 commented on May 30, 2026

VL4DST3R commented on May 30, 2026

Santelomar commented on Jun 10, 2026

github-actions commented on Aug 9, 2026

hadifarnoud commented on Oct 16, 2026

rekram1-node commented on Oct 16, 2026

rekram1-node commented on Oct 16, 2026

rekram1-node commented on Oct 16, 2026

rekram1-node commented on Oct 16, 2026

hadifarnoud commented on Oct 17, 2026

rekram1-node commented on Oct 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Root cause: the `modalities` field is required but undocumented

Workaround: declare `modalities` explicitly