Repository navigation
Working with REST & media data? [correct approaches?] #6665
Description
Question
New to OpenCode, testing with the Big Pickle model. Question: how are OpenCode LLMs expected to interact with image (media) data?
Test case: I rebuilt an MCP that accepts images in base64 data-URI format to OCR the configured image. I expected the system to understand that files are read into bytes and that bash can encode base64 — that does not appear to be the intended out-of-box behavior.
Searching for standard MCPs that are REST-API aware (standard conversions and conventions) returned no results; the MCP.org Discord channels appear inactive. If Playwright can drive a browser, one would expect a standard utility MCP for mapping between filesystems, LANs, media, and REST APIs. This is likely a knowledge gap on my side — please point me at the right pattern.
In the meantime I wrote a small MCP servlet that converts filenames to base64 data URIs (a few hundred lines, working). The next step exposed the failure mode below.
The LLM, asked to OCR twain.jpg, went through:
- Sending the filename as a URL stub to the OCR backend (malformed request) — fails.
- Reading the file directly (correct) — reports success, but the data is not usable downstream.
- Navigating the browser to
file://twain.jpgand taking a screenshot (via Playwright) to "OCR the screenshot". - Reading the file again, then attempting to run JavaScript inside the browser to base64-encode it (
require is not defined— fails). - Proposing to run Python on-device, but generating JavaScript and attempting to run it in the browser instead.
The concerning part for a new user: OpenCode appears to default-permit executing AI-generated JavaScript in the browser (Playwright evaluate/run_code), and would likely also permit AI-generated Python on-device. As a novice user I could not tell from the UI which permissions apply. The agent also appears unaware of its own data-type limitations (e.g., that the model cannot consume raw binary).
Questions:
- What is the intended pattern for passing local media files to MCP tools from an LLM?
- Is AI-generated code execution in the browser (and on-device) permissioned by default? If so, is that documented?
- Should an LLM be expected to know its own data-type limitations, and is there metadata for that?
Happy to provide full traces and the MCP servlet code on request.
Plugins
No response
OpenCode version
No response
Steps to reproduce
- Provide a local image file.
- Ask the agent to OCR it via an MCP tool that expects base64.
- Observe the failure cascade above.
Screenshot and/or share link
No response
Operating System
No response
Terminal
No response
Activity
github-actions commentedon Jan 2, 2026
This issue touches on several important areas that are already being tracked:
Image/Media Data Handling Issues:
LLM Hallucination & Tool Usage Errors:
Tool Execution & Security Concerns:
Your detailed walkthrough of the LLM's failure modes and the security concerns about executing AI-generated JavaScript/Python code are valuable insights that overlap with these existing issues. Please check those issues first to see if any address your specific use case, or feel free to cross-reference this issue in those discussions if you believe your findings add important context about error handling and hallucination patterns.
rekram1-node commentedon Jan 2, 2026
Can u show me what model u are using it seems like it doesnt support images or doesnt have images listed as a modality it supports?
rekram1-node commentedon Jan 2, 2026
Ah:
Yeah it doesnt support images
davidbernat commentedon Jan 2, 2026
@rekram1-node @fwang Thank you. That is news to me, and great insight; the model has not yet alerted me that images are not handled in the week of poking and prodding. Nonetheless, the question is a different scope than whether a specific model supports a specific datatype; plus if we pick nits the model continued to fail with a simple MCP mapping filename to the requested data, then escalated to incredibly creative bad responses. Not only would such a junior engineer be reprimanded to the point of firing, and certainly in a live demonstration, there are permission concerns up and down the stack; in that sense, the specific model Big Pickle not being aware of its own specific, fine-grain limitations is an additional concern to this ticket, not a solution. So, to boil the question down:
Thanks so much! You guys are on the precipice of Big Consultant glory! I seriously hope I get an invite to a few working groups!
github-actions commentedon May 13, 2026
To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.