Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

Allow plugins to customize behavior when a model doesn't support a media modality (image/audio/video)

Summary

When a user attaches a media file (image, audio, video, PDF) to a session using a model that doesn't support that modality, unsupportedParts() in packages/opencode/src/provider/transform.ts hardcodes the media part into a generic error text:

ERROR: Cannot read "filename.png" (this model does not support image input). Inform the user.

This is a dead-end for the LLM — it can only "inform the user" and cannot take any productive action. Plugin developers have no way to intercept this and guide the LLM toward an alternative path (e.g., calling a plugin-provided tool that recognizes the media via a separate multimodal model).

Use Case

I'm building a plugin that provides ae-image, ae-audio, and ae-video tools. These tools:

1. Accept a local file path as input
2. Read the file, detect MIME type via magic bytes
3. Send it to a separately-configured multimodal model (via modelScenarios.vision/audio/video in opencode.json)
4. Return a structured Markdown description of the media content

The intended flow when a user is on a text-only model (e.g., a coding model) and attaches an image:

User: "What's in this screenshot? @screenshot.png"
                    ↓
opencode detects model doesn't support image
                    ↓
Plugin injects guidance: "Image at /path/to/screenshot.png — call ae-image tool to recognize it"
                    ↓
LLM calls ae-image(file="/path/to/screenshot.png", prompt="...")
                    ↓
ae-image returns Markdown description
                    ↓
LLM now has the image content as text and continues working

This lets users work with media content even on text-only models, without manually switching models or running separate commands.

Current Limitations

1. No hook after unsupportedParts()

unsupportedParts() runs inside the AI SDK's transformParams middleware (packages/opencode/src/session/llm.ts:328-341), which is the last step before the HTTP request goes to the provider. There is no plugin hook that fires after this point. The available hooks are:

All plugin hooks fire before unsupportedParts(). A plugin cannot react to or customize the "unsupported media" behavior.

2. Pre-emptive degradation has gaps

Since plugins can only act before unsupportedParts(), they must pre-emptively convert media to paths. But this requires the plugin to know whether the model supports the modality — which introduces its own problems:

• Capability cache race conditions: The plugin calls client.provider.list() to cache model capabilities, but this is async and may not be ready when chat.message fires. When the cache misses, the plugin defaults to "assume supported" (to avoid unnecessary conversion), and the media reaches unsupportedParts(), producing the dead-end error.

• Unknown models: When a model isn't in the models.dev database, capabilities are unknown. The plugin has to guess — either too conservative (converts media even for capable models, forcing unnecessary tool calls) or too permissive (lets media through, hitting the dead-end error).

• Duplicated logic: The plugin must re-implement modality detection (mimeToModality), capability lookup, and the decision tree — all of which opencode already does in unsupportedParts(). This is fragile and will drift if opencode's logic changes.

3. The error text doesn't guide the LLM

Even when unsupportedParts() fires correctly, the text ERROR: Cannot read ... Inform the user. gives the LLM no actionable path. A plugin may have registered a tool (e.g., ae-image) that could solve this, but the LLM isn't told about it. The LLM just reports the error to the user, and the user has to manually intervene.

Proposed Solutions

I'd appreciate any one of the following approaches. Listed in order of preference:

Option A: Hook after unsupportedParts (preferred)

Add a new hook, e.g. experimental.provider.unsupported_media, that fires after unsupportedParts() converts a media part to the error text. The hook receives:

"experimental.provider.unsupported_media"?: (
  input: {
    sessionID: string
    model: Model
    parts: Array<{ modality: Modality; filename?: string; mime: string; originalPart: ContentPart }>
  },
  output: { replacements: Array<{ text: string }> },
) => Promise<void>

The plugin can replace the hardcoded error text with a custom message that guides the LLM to call a tool:

Image file at /path/to/screenshot.png. The current model does not support image input.
Call the `ae-image` tool with `file="/path/to/screenshot.png"` to recognize its content.

Option B: Make unsupportedParts replacement text configurable

Allow plugins (or agent/model config) to register a custom replacement text template for unsupported media. For example, in opencode.json:

{
  "agent": {
    "my-agent": {
      "unsupportedMediaText": "Cannot read {filename} ({modality} not supported). Call the ae-{modality} tool with file=\"{path}\" to recognize it."
    }
  }
}

Or via a plugin hook that provides the template, with {filename}, {modality}, {path} placeholders.

Option C: Pass file path to the replacement text

At minimum, include the file path (not just the filename) in the error text, so the LLM has enough information to call a tool if one is available:

Current:

ERROR: Cannot read "screenshot.png" (this model does not support image input). Inform the user.

Improved:

Cannot read "screenshot.png" (this model does not support image input). File path: /absolute/path/to/screenshot.png

This alone would let a plugin's system prompt instruct the LLM to call ae-image with that path, without needing a new hook.

Environment

• opencode version: latest main branch
• Relevant files:
  • packages/opencode/src/provider/transform.ts:374-410 (unsupportedParts)
  • packages/opencode/src/session/llm.ts:325-342 (transformParams middleware)
  • packages/plugin/src/index.ts:282-290 (experimental.chat.messages.transform hook)

Notes

• Option A is the most flexible — it lets plugins fully customize the behavior per-modality, per-model, and even inject different guidance based on which tools are registered. The originalPart field preserves the file URL/path so plugins can extract it.
• Option B is simpler to implement and covers the common case, but doesn't allow dynamic logic (e.g., "only guide the LLM to ae-image if that tool is enabled for this session").
• Option C is the minimal change — just adding the path to the existing text — but it depends on the LLM being smart enough to act on it, and doesn't remove the "Inform the user" dead-end phrasing.

Happy to contribute a PR if any of these directions sound reasonable. Thanks for the great work on opencode!