Responses image parts omit detail, 400 on strict upstreams #59404

Activity

1. opencode-agent commented on Oct 26, 2026

   

   opencode-agentBot

   on Oct 26, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report. I was able to reproduce this on 2.1.4, and it still happens on the latest dev (161d4669).

   Setup:

   • A custom provider with package @opencode/ai/providers/openai-compatible/responses, pointed at a small local Responses server.
   • That server checks every input_image part the same way vLLM does: pydantic validation against openai-python's ResponseInputImageParam.
   • The model asks the agent to read a PNG.

   The follow-up request sends the PNG in the tool result as {"type":"input_image","image_url":"data:image/png;base64,..."} with no detail key. Validation fails with 1 validation error for ResponseInputImageParam / detail / Field required, and the run ends with that error. In lowerMedia (packages/ai/src/protocols/open-responses.ts), detail is only set when provider metadata supplies one, and image-detail.test.ts currently expects it to be left out in that case.

   The script below reproduces it end to end. It needs Python with openai and pydantic, and can run either the installed opencode or a source checkout via OC_DEV.

   Reproduction script (repro.sh)

   #!/usr/bin/env bash
   # Reproduces anomalyco/opencode#59404: Responses input_image parts sent without `detail`.
   # A tiny Responses server (embedded below) validates every input_image with
   # openai-python's ResponseInputImageParam via pydantic, as vLLM does, and returns 400 on failure.
   # Needs: python3 with `openai` + `pydantic`, and `opencode` on PATH (or OC_DEV=/path/to/checkout).
   set -euo pipefail
   PY=${PY:-python3}
   PROJ=$(mktemp -d); export HOME=$(mktemp -d); export XDG_CONFIG_HOME=$HOME/.config XDG_DATA_HOME=$HOME/.local/share
   "$PY" - "$PROJ/image.png" <<'EOF'
   import sys,zlib,struct
   def c(t,d): return struct.pack(">I",len(d))+t+d+struct.pack(">I",zlib.crc32(t+d)&0xffffffff)
   raw=b"".join(b"\x00"+b"\xff\x00\x00"*4 for _ in range(4))
   open(sys.argv[1],"wb").write(b"\x89PNG\r\n\x1a\n"+c(b"IHDR",struct.pack(">IIBBBBB",4,4,8,2,0,0,0))+c(b"IDAT",zlib.compress(raw))+c(b"IEND",b""))
   EOF
   cat > "$PROJ/opencode.json" <<'EOF'
   {
     "providers": {
       "vllm": {
         "package": "@opencode/ai/providers/openai-compatible/responses",
         "settings": { "baseURL": "http://127.0.0.1:8765/v1", "apiKey": "x" },
         "models": { "qwen": { "capabilities": { "tools": true, "input": ["text", "image"], "output": ["text"] } } }
       }
     },
     "model": "vllm/qwen"
   }
   EOF
   cat > "$PROJ/server.py" <<'PYEOF'
   # Minimal Responses API server that validates input image parts the way vLLM does:
   # pydantic TypeAdapter over openai-python's ResponseInputImageParam (detail is Required).
   # Turn 1: asks the agent to call `read` on image.png. Turn 2+: validates the tool result images.
   import json, sys
   from http.server import BaseHTTPRequestHandler, HTTPServer
   from pydantic import TypeAdapter, ValidationError
   from openai.types.responses.response_input_image_param import ResponseInputImageParam
   
   adapter = TypeAdapter(ResponseInputImageParam)
   LOG = open("requests.jsonl", "a")
   
   def images(node):
       if isinstance(node, dict):
           if node.get("type") == "input_image":
               yield node
           for v in node.values():
               yield from images(v)
       elif isinstance(node, list):
           for v in node:
               yield from images(v)
   
   def sse(events):
       return "".join(f"event: {e['type']}\ndata: {json.dumps(e)}\n\n" for e in events).encode()
   
   class H(BaseHTTPRequestHandler):
       def do_POST(self):
           body = json.loads(self.rfile.read(int(self.headers["content-length"])))
           LOG.write(json.dumps(body) + "\n"); LOG.flush()
           imgs = list(images(body.get("input")))
           for img in imgs:
               try:
                   adapter.validate_python({k: v for k, v in img.items()}, strict=False)
                   if "detail" not in img:
                       raise ValueError("detail: Field required")
               except (ValidationError, ValueError) as e:
                   shown = {k: (v[:40] + "...") if isinstance(v, str) and len(v) > 40 else v for k, v in img.items()}
                   print("REJECTED input_image:", shown, "->", str(e).splitlines()[0], flush=True)
                   msg = json.dumps({"error": {"message": f"1 validation error for ResponseInputImageParam\ndetail\n  Field required", "type": "BadRequestError", "code": 400}}).encode()
                   self.send_response(400); self.send_header("content-type", "application/json")
                   self.send_header("content-length", str(len(msg))); self.end_headers(); self.wfile.write(msg); return
           has_tool_output = any(i.get("type") == "function_call_output" for i in body.get("input", []) if isinstance(i, dict))
           print(f"request: {len(imgs)} input_image part(s), tool output: {has_tool_output}", flush=True)
           if not has_tool_output:
               item = {"type": "function_call", "id": "fc_1", "call_id": "call_1", "name": "read",
                       "arguments": json.dumps({"path": "image.png"}), "status": "completed"}
           else:
               item = {"type": "message", "id": "msg_1", "role": "assistant", "status": "completed",
                       "content": [{"type": "output_text", "text": "I see the image.", "annotations": []}]}
           resp = {"id": "resp_1", "object": "response", "model": body.get("model"), "status": "completed", "output": [item],
                   "usage": {"input_tokens": 10, "output_tokens": 5, "total_tokens": 15}}
           ev = [{"type": "response.created", "response": {**resp, "status": "in_progress", "output": []}},
                 {"type": "response.output_item.added", "output_index": 0, "item": item},
                 {"type": "response.output_item.done", "output_index": 0, "item": item},
                 {"type": "response.completed", "response": resp}]
           if item["type"] == "message":
               ev.insert(2, {"type": "response.output_text.delta", "item_id": "msg_1", "output_index": 0, "content_index": 0, "delta": "I see the image."})
           data = sse(ev)
           self.send_response(200); self.send_header("content-type", "text/event-stream")
           self.send_header("content-length", str(len(data))); self.end_headers(); self.wfile.write(data)
       def log_message(self, *a): pass
   
   HTTPServer(("127.0.0.1", int(sys.argv[1]) if len(sys.argv) > 1 else 8765), H).serve_forever()
   PYEOF
   cd "$PROJ"; "$PY" server.py 8765 > "$PROJ/server.log" 2>&1 & SERVER=$!; trap 'kill $SERVER' EXIT; sleep 1
   if [ -n "${OC_DEV:-}" ]; then
     # From source, --standalone's spawned server misses --conditions=browser, so start `serve` explicitly.
     OC=(bun --conditions=browser --preload="$OC_DEV/packages/tui/node_modules/@opentui/solid/scripts/preload.js" "$OC_DEV/packages/cli/src/index.ts")
     "${OC[@]}" serve --port 4097 > serve.log 2>&1 & OCS=$!; trap 'kill $SERVER $OCS' EXIT
     until grep -q "server password" serve.log; do sleep 1; done
     export OPENCODE_SERVER_PASSWORD=$(awk '/server password/{print $3}' serve.log)
     CONNECT=(--server http://127.0.0.1:4097)
   else OC=(opencode); CONNECT=(--standalone); fi
   "${OC[@]}" --version
   timeout 180 "${OC[@]}" run "${CONNECT[@]}" --auto -m vllm/qwen "Read image.png" 2>&1 | tail -5 || true
   echo "--- server log"; cat server.log

   Run it: bash repro.sh
2. opencode-agent

   added

   reproduced

   on Oct 26, 2026
3. Dante-dan commented on Oct 26, 2026

   

   Dante-dan

   on Oct 26, 2026

   Contributor

   More actions

   I am taking a look at this. I will default missing image detail to auto in the shared Responses lowering path, while preserving explicit detail values and file handling, and update the existing image/tool-result tests.