Groq Kimi K2 Caching

This issue might be a duplicate of existing issues. Please check:

groq + kimi k2: Failed to call a function #1018: Similar Groq + Kimi K2 combination with function calling failures
Groq: Models return error #222: General Groq model errors with tool execution
Groq Kimi K2 Instruct: tool call validation failed: parameters for tool edit did not match schema #1446: Specific Groq Kimi K2 Instruct tool call validation issues

Feel free to ignore if none of these address your specific case.

From groq:

Prompt caching works automatically on all your API requests with no code changes required and no additional fees.

But ig you are saying to track the cached tokens?

Ha, I'd not noticed that.
Tracking cached tokens info returned by the API would be ideal of course for accurate pricing estimate.
Lower priority perhaps though, feel free to close.

I will leave open because we may as well track it properly if we aren't

Does the prompt caching work correctly for others? It seems Groq is still billing me the full cost for the total number of input tokens for each request in a session without considering the cache.

It is unfortunate they don't show this information in their dashboard logs.

@djmaze

Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually

@djmaze

Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually

@djmaze

Yes, I indeed used Kimi K2 for testing this.

So - this is a bit longer, but maybe helpful

Spoiler for attention the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs BUT the real cost might differ, because of the ~235k tokens which will be invoiced by groq

Update: The cost actually doesn't differ too much in my test - the dashboard shows me $0.16 for today

I recently contacted groq because a session blew up to 10mio tokens (and I'm pretty sure that the session would have been 1-2mio on moonshot directly).

They explained me that there might be a difference because other providers might not charge cached tokens or make it wayyy cheaper.

I still had the impression something fishy is going on ... and since i love the speed of groq - and think kimi is fantastic i invested some time to track down my issue.

TL;DR

I think that for some reason either opencode or groq add some "overhead" tokens for each request (between 10k and 30k), which because of the cached token cost of groq sooner or later get a bit expensive.
Those 10-30k might not seem a lot, but considering that opencode can work in a multi-step agentic way (f.e. calling 10 times the llm for 1 user input) - those tokens add up quickly (see my intro - 10mio)
This overhead seems smaller when calling moonshot.ai (same model)
- first request for java hello world ~ 10k vs. ~ 20k
- hello world in python with some text changes - ~10.4k
Note the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs BUT the real cost might differ, because of the ~235k tokens which will be invoiced by groq

Here are my notes from my test:

Opencode session starts at ~11k tokens

directly after asking for a hello world in python opencode shows 11k
i don't understand why - neither the system prompt is that big, nor my request to "write a hello world python script" command
- i did check anthropic.txt with https://tokencounter.org/
- i did check gemini.txt with https://tokencounter.org/
- i did check copilot-gpt-5.txt with https://tokencounter.org/
for the second request (change the hello world to something else) - the token usage was again ~10k ... which ... i don't understand as well
starting from there it kept somewhat reasonably sized

Groq Dashboard shows above input tokens

groq counts the 11k from the first call
groq counts the 10k from the second call
i started with ~8k so from here - i was at +/- 30k tokens
those 30k tokens pay dividends

Caching confusion

second call

during the second call - groq counts the 11k from the first call as cached
plus 10k not cache hit

third call

cache when i started 8k plus
already first call cache 11k plus
prev. non cached 10k becoming addon cache plus
800 new input (no cache hit tokens)

fourth .. n-th call

already 30k cached + n-1 call = new cache
minimum sized token as input cache
Cache after last call in this session: 205,3k
Uncached input after last call in this session: 34,3k
Total token count: 241,2k

NEW(!) Session

started a new session with /new expecting to start with "0" for the session
asked for a hello world in java
this time the request caused a ~22k token count (see screen of opencode below)
groq's uncached input token count reports for today 34,8k (so ~500 tokens new)
Cached Input Tokens: 226,8k (so ~21k cached)

new session +1

this request changed the Opencode counter from 21,7k to 22,1k ... so ~400 tokens
groq's uncached input token count reports: 35,1k (so ~300 new tokens added)
groq's cached input token count reports: 248,6k (so ~22k tokens added)

Baseline (I started with some tokens already used)

First python request

Second python request

3rd python request

4th python request

5th python request

6th python request

7th python request

Opening new session asking for java

2nd java request

I thought grok caches automatically? And if so isn't this a problem on their end?

it does - the big question is "where do those 20k tokens come from with each request" which (yes) gets cached, but still blow up the token count

if i ask "please write hello world in python" - this should not have 22k input token

if i ask "change the string from hello world to hello github" - this should not have 22k input token

it's not a "does groq cache" question - it does and i prove it in the test ... i think the main concern of OP (and me) are why do i still pay more than expected, and my test shows, that with each call there is some overhead constantly transferred (or magically added by groq)

and if my request in openspec triggers 10 sub steps - the overhead is multiplied by 10 - so there are 200k tokens cached (and invoiced with $0.33 per mio) ... spending 2-3 hours on a bigger change might get a bit painful

@kajdo

@kajdo well there is the system prompt + tools and their descriptions

in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback)

Also any mcp servers you may have

All those together generally come out to ~11-20k tokens depending on your setup.

The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for

@kajdo well there is the system prompt + tools and their descriptions

in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback)

Also any mcp servers you may have

All those together generally come out to ~11-20k tokens depending on your setup.

The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for

i just checked the difference between calling with Build mode vs. plan mode (only asking "how would a hello world in python look like")

Build Mode (tools activated but not used): 23.7k
Plan Mode (tools disabled): 11.8k

so you're definitely right, that the tools activation alone makes a big difference

assuming you're right with AGENTS.md as well - I guess that's the answer

not too happy, since it looks like i have to accept to burn more tokens :) ... but at least it makes more sense to me now (drove me crazy tbh) - thank you!

Yeah ofc!

Navigation Menu

Repository navigation

Groq Kimi K2 Caching #2174

TL;DR

Opencode session starts at ~11k tokens

Groq Dashboard shows above input tokens