SDK 0.3.0 does not support images in tool results' `binaryResultsForLlm`

[connor4312]

1. Define a tool that provides an image to the SDK
2. Have the agent call it in a model that supports it (e.g. Haiku)
3. 🐛 the agent can't see the image

Specifically for us this happens here: https://github.com/microsoft/vscode/blob/a1803b2a0c88b4a22ff83262b132cca1befc0886/src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts#L592

This is the image being returned: image.zip

Here is our log dump which includes the copilot CLI events.jsonl. You can see the binary results in that log:

ah-logs-Open-apple.com,-take-a-screenshot,-and-t.zip

And just for edification, here is the exact image that was returned, a valid jpeg: image.zip

Upstream issue: microsoft/vscode#315947

[stephentoub]

@connor4312, the linked code is using type: c.contentType... doesn't that mean for a png it's going to be setting the type to "image/png" rather than "image"? The runtime expects "image"; the mimeType is separate.

[connor4312]

Ah, yes, can confirm that works. It looks like the alternate type is resource based on what I see in the runtime. I'll leave this issue open if you want to track documentation update -- I don't think we document anything about this in the TS SDK at least

[stephentoub]

Thanks, @connor4312. I'm looking at changing it to be an enum so that it's self documenting.

[stephentoub]

Addressed. Thanks.