Repository navigation
[FEATURE]: Auto-describe images via vision fallback when active model lacks vision support #24948
Description
Feature hasn't been suggested before.
- I have verified this feature I'm about to request hasn't been suggested before. (See note below about [FEATURE]: Auto image-to-text transcription via multimodal model for non-multimodal providers #22828.)
Describe the enhancement you want to request
When using a coding-focused LLM that doesn't support image input (e.g. DeepSeek, GLM, Haiku), pasting a screenshot into the chat causes the image to be silently dropped. The model receives an error string instead, with no indication that images aren't supported.
Proposed solution: Before calling streamText, if the active model can't read images but the message contains image parts, automatically find a vision-capable model from any configured provider, call it with a description prompt, and replace the image parts with the returned text description. The main model then receives the description as plain text — entirely transparent, no model switching required.
A new optional vision_model config field lets users pin a specific model (e.g. openai/gpt-4o). If not set, the first image-capable model found across all configured providers is used, skipping the current model to handle cases where the active provider has billing or rate-limit issues.
Implementation: PR #24382
Note on #22828: That issue proposes a similar concept but with a different scope — it focuses on transcription for non-multimodal providers, while this proposal handles the case where the specific active model lacks vision support, regardless of provider, and includes fallback to any available provider. The implementation here also adds a dedicated vision_model config field and integrates into the streaming pipeline rather than as a pre-processing step.
Activity
github-actions commentedon Apr 29, 2026
This issue doesn't fully meet our contributing guidelines.
What needs to be fixed:
[FEATURE]:and include the required verification checkboxes.Please edit this issue to address the above within 2 hours, or it will be automatically closed.
This issue might be a duplicate of existing issues. Please check:
vision_model/image_transcription_modelconfig optionIf you believe this was flagged incorrectly, please let a maintainer know.
[-]Non-vision models silently drop image attachments from chat[/-][+]Auto-describe images via vision fallback when active model lacks vision support[/+]github-actions commentedon Apr 29, 2026
Thanks for updating the issue! It's much improved and has real, helpful content. However, there is still one compliance issue to address:
Missing title prefix: Feature Request issues must have a title starting with
[FEATURE]:. Please update the title to:Once that's updated, we'll remove the
needs:compliancelabel.[-]Auto-describe images via vision fallback when active model lacks vision support[/-][+][FEATURE]: Auto-describe images via vision fallback when active model lacks vision support[/+]github-actions commentedon Apr 29, 2026
Thanks for updating the issue! Everything looks good now — the feature request is clear, well-scoped, and follows the contributing guidelines. Removing the compliance flag.
github-actions commentedon Jul 4, 2026
To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.