Repository navigation
When calling to vllm server with hermes tool parser opencode doesn't call the tools #1122
Description
I'm trying to get this working with my local vllm server on fedora 41. Here's the current server command I'm running:
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --tensor-parallel-size 4 --enable-auto-tool-choice --tool-call-parser hermes --gpu-memory-utilization 0.8 --max-model-len 30000 --trust-remote-code
It seems that the model generates the tool calls fine, but but opencode just shows them as text instead of calling them. Is support for this feature coming later? Am I missing an easy workaround?
Activity
Anodyine commentedon Jul 18, 2025
I actually got it working with a local model that is served with vllm 0.9.2. It took a few messages to get it to use the tools, but eventually it made an edit that worked. So at least it proves that the tool calling can work with a local model for opencode 0.3.20. However, it is not practical for use yet. For more complex tasks it just thinks and then doesn't respond. I wonder if it is timing out. I would appreciate any help and will be looking out for updates that help us use local models better.
Here is the vllm serve command that worked on my system with 4 3090 gpus:
vllm serve Qwen/Qwen3-32B --enable-reasoning --reasoning-parser deepseek_r1 --tensor-parallel-size 4 --gpu-memory-utilization 0.8 --max-model-len 20000 --enable-auto-tool-choice --tool-call-parser hermes --trust-remote-code
peaster commentedon Aug 8, 2025
@Anodyine did you have any further progress with this? I'm trying to do the same with Qwen3 30B Coder on dual 3090s and i'm seeing roughly the same issues. It just doesn't process tool calls
Anodyine commentedon Aug 19, 2025
@peaster Yeah actually I had some great success with vllm 10.0 since it supports the qwen3_coder tool parser. I used the qwen cli client and it is very fast and effective.
Here is the relevant article:
https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-Coder-480B-A35B.html
Of course, I was using this model not the 480B one the article mentions.
https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
Anodyine commentedon Aug 19, 2025
@peaster Here is my start server command for reference. You need to reduce tensor-parallel-size to 2 and may need to reduce max model length and/or gpu-memory-utilization if you run out of memory.
/opt/vllm/.venv/bin/vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct
--tensor-parallel-size 4
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--gpu-memory-utilization 0.9
--max-model-len 200000
--max-num-seqs 2
--max-num-batched-tokens 16384
--enable-chunked-prefill
--enable-prefix-caching
--swap-space 32
--trust-remote-code
--host 127.0.0.1
--port 8000
peaster commentedon Aug 19, 2025
Thanks! i did get it mostly working using the new tool call parser. In opencode specifically, it's struggling on a couple of the tools though. I commented in this thread and that mostly works #1809 (comment) but it would be nice to have this working natively too
loligapta commentedon Oct 15, 2025
Sir, how did you connect opencode to local running vLLM server? can share config please sir?
feat(vllm): switch from Gemma to Qwen3-Coder for better tool calling …
zaidorx commentedon Jan 20, 2026
@loligapta Not sure if this is still relevant but I recently discovered and set up opencode with vllm and here is the section of my config.json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"local-vllm": {
"npm": "@ai-sdk/openai-compatible",
"name": "vLLM (Local)",
"options": {
"baseURL": "http://192.168.99.23:8000/v1",
"apiKey": "dummy-key"
},
"models": {
"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {
"name": "Qwen3-Coder-30B"
}
}
}
}