Add metadata awareness and data type validation/blacklists for LLM tool execution #10820

Description

What are we missing here about reliability standards in this industry? Thanks.
This is a blocker before adopting LLMs for client use re: maintenance of these problems goes exponential.

Summary

OpenCode currently lacks awareness of LLM metadata (names, limitations, supported data types) and doesn't validate tool inputs/outputs against LLM capabilities. This causes session-breaking failures when tools return unsupported data types, particularly for binary data like images.

Problem Description

1. No Metadata Awareness

• OpenCode doesn't know LLM names (e.g., "BigPickle")
• OpenCode isn't aware of LLM data type limitations (i.e., BigPickle will interpret base64 encoded image data as text prompts, and begin spooling out similar gibberish)
• No capability matching between tools and LLMs (i.e., LLMs cannot be informed to manually override its choice of a catastrophic tool)

2. No Data Type Validation

• Tools can execute without checking if LLM can handle the output
• No validation that an LLM can handle tool return types prior to execution
• e.g. Binary data can be passed to text-only LLMs

3. Session-Breaking Failures

• Binary image data causes catastrophic failures
• Sessions end abruptly and require restart
• No graceful degradation or error handling

4. No Manual Control of Limitations

• MCP method blacklist to disallow breaking methods on third-party MCPs
• (Advanced) No hooks for manual processing of data (i.e., catch an MCP returning bytes data with a bespoke handler to write to disk, return locla filename instead)

Use Cases & Examples

Playwright MCP Screenshot Scenario

1. User requests: "Take a screenshot of this URL"
2. OpenCode correctly identifies and executes Playwright MCP screenshot method
3. MCP returns binary image data
4. BigPickle receives binary data, treats it as text/JSON prompt
5. Session crashes and requires restart, smoke emerges from computer, lightning crashes in distance, evil released from the depths Barad-dûr (typical)

OCR MCP Server Scenario

1. User wants to OCR an image using MCP server
2. API expects base64 encoded data URIs
3. LLM cannot handle binary image data directly
4. OpenCode cannot be provided a hook to execute beforehand; to modify MCP interface to accept filename
5. Requires custom self-owned wrapper to handle file conversion; cannot "turn off" dangerous MCP methods

Proposed Solutions

Solution 1: Metadata Awareness

{
  "llm_metadata": {
    "name": "BigPickle",
    "capabilities": ["text/plain", "text/json"],
    // simple
    "tool_blacklists": {
        "playwright": ["browser_take_screenshot"]
    }
    // sensible, though will get numerous quickly
    "tool_replacements": {
        "playwright": {
            "browser_take_screenshot": "my_alternative_version"
        }
    }
    // advanced
    "tool_handlers": {
        "playwright": {
            "browser_take_screenshot": "my_handlers.py:ClassName"
        }    
    }
    "max_tokens": 50000
  }
}

Solution 2: Pre-Execution Validation

• Check tool return types against LLM capabilities
• Block incompatible tool executions (from being selected)
• Provide user-friendly error messages (if no other solution)
  • "[Thinking]The Playwright MCP has a browser-take-screenshot method but BigPickle cannot use its image data return value.[/Thinking] It looks like Playwright MCP is supposed to handle this situation but as BigPickle I lack this functionality. Would you like to switch to a different LLM or should we write session code to handle these datatypes manually?"

Solution 3: Automatic Wrapper Generation

• Detect incompatible tool/LLM combinations
• OpenCode create wrapper methods automatically
• Handle data type conversions transparently

Current Workarounds

Users must build and support custom MCP wrappers that:

1. Execute the original MCP method
2. Handle incompatible data types (save to disk, convert, etc.)
3. Return only JSON-compatible results
4. Clean up temporary files

This overhead is because of the limitation of the MCP Protocol and the lack of transparency into the OpenCode tool use, which makes this particularly suitable for resolution for the OpenCode company.

Impact & Rationale

Why This Matters

• Session Reliability: Prevents catastrophic failures
• User Experience: Eliminates confusing crashes
• Developer Productivity: No need for high volume of monkeypatch wrappers, which must be supported for each MCP method, by each developer.
• Company Growth: Makes OpenCode more robust, and in line with consumer expectations.

Additional Context

This issue represents a fundamental gap in LLM tool orchestration. While OpenCode correctly identifies and executes tools, it lacks the awareness to ensure tool outputs are compatible with the executing LLM. The current approach puts the burden on users to create custom workarounds for what should be a platform-level concern.

The fact that this can be solved with relatively simple metadata awareness and validation rules, against an infinite variety of monkeypatches supported by third-party users, makes it an ideal candidate for immediate implementation.

Thanks for handling the request this text is just lax writing as my brain is up with the sun here and still groggy. Using OpenCode v1.1.36.