[URGENT] Go: frank/DeepSeek-V4.1-Flash upstream repeatedly drops prompt cache — ~50% of billed usage is full-context re-reads #50258

Activity

1. Git-yzc commented on Sep 21, 2026

   

   Git-yzc

   on Sep 21, 2026

   More actions

   Independent corroboration: cache loss reproduces with raw HTTP, on a different client and OS

   Confirming this from a different client (DeepSeek Harness 2.0.13, Windows 11) against the same endpoint https://opencode.ai/zen/go/v1/chat/completions, model deepseek-v4.1-flash. The losses are not client-side prompt construction, and they reproduce without any agent involved.

   1. Byte-identical requests flap between full hit and full miss

   Same payload, same headers, same x-opencode-session, sent sequentially:

   #1   2.12s  prompt=6019  cached=0     hit=  0.0%
   #2   6.35s  prompt=6019  cached=0     hit=  0.0%
   #3   3.33s  prompt=6019  cached=0     hit=  0.0%
   #4   3.25s  prompt=6019  cached=0     hit=  0.0%
   #5   2.20s  prompt=6019  cached=6016  hit=100.0%
   #6-#12                                  hit=100.0%   (8 consecutive hits)
   #13  2.86s  prompt=6019  cached=0     hit=  0.0%
   #14  2.85s  prompt=6019  cached=6016  hit=100.0%
   #15  4.04s  prompt=6019  cached=0     hit=  0.0%
   #16  2.96s  prompt=6019  cached=6016  hit=100.0%
   #17  2.51s  prompt=6019  cached=0     hit=  0.0%
   #18  1.72s  prompt=6019  cached=0     hit=  0.0%
   
   => 8/18 cold misses (44%), mean hit 55.5%
   

   Nothing in the request differs between those calls, so no client-side cache key, header, or prompt handling can account for it. A second run 20 minutes later gave 1/12 — the rate varies with time, which points at upstream replica/eviction behavior rather than a deterministic bug.

   2. Session id does influence locality, but does not provide stickiness

   Alternating two x-opencode-session values across 12 identical requests gave 3/12 cold misses, versus 1/12 when repeating a single id. So the header is being used for something, but it is clearly not enough to keep a prefix warm by itself.

   3. Regression timeline from client session records

   From 8,150 assistant turns of local session data (per-step inputTokens / cacheReadTokens; "mid-session" excludes the first two steps of a session, which are always cold):

   date     mid-session cache hit   full-context re-reads
   09-13    99.7%                   0
   09-14    99.7%                   1
   09-15    99.5%                   0
   09-16    99.6%                   1
   09-17    99.6%                   0
   09-18    99.2%                   1
   09-19    99.0%                  15  (1%)
   09-20    91.8%                  62  (6%)   <-- regression starts
   09-21    79.3%                  32  (16%)
   

   Median per-step latency moved with it: 8.0s (09-16) → 13.8s (09-20) → 13.5s (09-21).

   The prompt prefix itself is stable — most steps add only 43–136 uncached tokens while the prompt grows monotonically:

   prompt=54083  new=451   cached=53632  hit=99.2%
   prompt=55339  new=43    cached=55296  hit=99.9%
   prompt=65260  new=23276 cached=41984  hit=64.3%   <-- drops for no reason
   prompt=88063  new=88063 cached=0      hit= 0.0%   <-- full re-read
   prompt=88272  new=1488  cached=86784  hit=98.3%   <-- recovers
   

   So the server is dropping a warm prefix, not the client rewriting history. Note the shape is flapping, not a one-way break — which is what per-upstream cache pools without affinity would produce.

   4. Other models on the same endpoint (6 identical requests each, same run)

   model	cold misses	note
   deepseek-v4.1-flash	1/6	stable once warm
   deepseek-v4-flash	1/6	stable once warm
   glm-5.3-flash	6/6	no caching at all
   minimax-m3	0/6	flaps between 128 and 3072 cached

   Consistent with your frank finding that this is upstream-specific rather than model-wide.

   5. Client note, in case it affects affinity routing

   This client currently sends no per-conversation session id on the adapter path used here (DeepSeek Harness is listed under "Known Problematic Clients" on the Go docs). It is worked around by configuring a fixed x-opencode-session value for all conversations, so every conversation shares one routing key. Flagging it since it is the only session signal the gateway receives from us — though given (1), it cannot be the whole story.

   Happy to share the repro script or the raw session records.

   👍React with 👍1Reacted by geminigeek
2. Veld101 commented on Sep 21, 2026

   

   Veld101

   on Sep 21, 2026

   More actions

   I looked into the caching code and here is what I found:

   Code Analysis

   V1 path (packages/opencode/src/provider/transform.ts:1310-1323)

   The promptCacheKey is only set for specific SDK packages:

   • @ai-sdk/deepinfra / @ai-sdk/cerebras → prompt_cache_key (snake_case)
   • @ai-sdk/openai / @ai-sdk/azure / @ai-sdk/xai / @ai-sdk/mistral / venice-ai-sdk-provider → promptCacheKey (camelCase)

   DeepSeek uses @ai-sdk/openai-compatible, which is NOT in the list. So the V1 path does not send any cache key to DeepSeek by default.

   V2 path (packages/core/src/session/runner/llm.ts:204-214)

   const promptCacheKey = /^ses_[0-9a-f]{64}$/.test(session.id) ? session.id.slice(4) : session.id
   providerOptions: { openai: { promptCacheKey } },

   V2 sets promptCacheKey under the openai provider options namespace, which may not be forwarded to DeepSeek-compatible endpoints.

   DeepSeek Cache Mechanism

   DeepSeek prompt cache is content-prefix-based, not key-based:

   • Consecutive requests sharing the same token prefix hit the cache
   • The cache granularity is token prefix matching, not session ID
   • Even a small change in the early part of the context (e.g., a system prompt change, a new tool result inserted) invalidates the downstream cache

   So prompt_cache_key is actually irrelevant for DeepSeek — the cache is content-addressed.

   The frank Node Problem

   Comparing the three upstreams from the reporter data:

   Upstream	Cache Hit Rate	Full Re-reads
   frank/DeepSeek-V4.1-Flash	94.2%	14
   deepseek/deepseek-v4.1-flash	97.2%	1
   deepseek-flash	95.9%	2 (idle TTL)

   The frank node has significantly worse cache performance. This strongly suggests frank is a replica pool without shared cache, and requests are being routed to different replicas without cache affinity.

   The mid-session cache drops (seconds/minutes after a normal hit, not idle TTL) are consistent with a load balancer rotating requests across replicas that each maintain independent caches.

   Conclusion

   This is a gateway routing / infrastructure issue, not a client-side code bug. The fix needs to happen at the frank upstream level:

   1. Ensure cache-affinity routing (sticky sessions based on session ID or content prefix)
   2. Or use a shared cache layer across replicas
   3. Or route traffic to a single replica per session

   The client-side prompt_cache_key cannot solve this because DeepSeek cache is content-addressed, not key-addressed.

   👍React with 👍1Reacted by geminigeek
3. Veld101 commented on Sep 23, 2026

   

   Veld101

   on Sep 23, 2026

   More actions

   Re-checked the current v2 code and two parts of my analysis above need correcting.

   1. The V2 request shape has moved and is not what I described.

   It is no longer providerOptions.openai.promptCacheKey in runner/llm.ts. On v2 today:

   • packages/core/src/session/model-request.ts:237-252 sets a first-class promptCacheKey from affinity = session.parentID ?? session.fork?.sessionID ?? session.id, and always sends header affinity on every request: x-session-affinity, X-Session-Id, x-parent-session-id, x-opencode-session.
   • packages/ai/src/protocols/openai-chat.ts:763-779 only adds the body field prompt_cache_key when model.compatibility.supportsPromptCacheKey === true (default off). The comment there notes header-based affinity is unaffected by that gate.
   • packages/core/src/plugin/provider/prompt-cache-key.ts flips that flag on for a whitelist of providers. The OpenCode Go route is not in it, so its body key stays off unless a user opts in.

   So the client already sends session-header affinity unconditionally. Enabling supportsPromptCacheKey (as the report did at 17:18) adds the body key on top — and the losses continued. That is consistent with the gateway not using either signal for sticky routing, rather than the client omitting a key.

   2. The V1 note still holds. The providerID.startsWith("opencode") branch in transform.ts is nested inside the api.id.includes("gpt-5") block, so DeepSeek never reaches it, and the package list I quoted is accurate.

   On the question above — detect the drop and fail or back off instead of paying for the re-read:

   There is no client-side action that avoids the charge.

   • A cache miss is not an error. The call returns 200 and is billed at the uncached rate; the client only sees cacheRead in the usage after the fact.
   • Retrying or failing does not help: the DeepSeek cache is content-prefix addressed, so the same prompt on the same route misses again. A retry only risks paying for the same full re-read twice.
   • The only lever is observability — surface a per-upstream cache-hit drop so the user can see it. It cannot reduce tokens already billed.

   The conclusion is unchanged: this has to be fixed upstream (cache affinity across replicas, or a shared cache layer), not in the client.

   👍React with 👍1Reacted by geminigeek
4. geminigeek commented on Sep 23, 2026

   

   geminigeek

   on Sep 23, 2026

   More actions

   this need to be fixed with priority , just resumed my GO after 2 month , this issue is crazy
5. geminigeek commented on Sep 23, 2026

   

   geminigeek

   on Sep 23, 2026

   More actions

   using legacy api key always route to direct deepseek endpoint , this is the workaround i found
6. zhengkaics commented on Sep 26, 2026

   

   zhengkaics

   on Sep 26, 2026

   Author

   More actions

   Update 09-26: cache losses now measurably burning the Go $60 Flash credit at full input price

   36-hour measurement from my own account. All 8 rows are the same session, prefix growing monotonically (586k → 595k tokens), model opencode-go/deepseek-v4.1-flash. cached from local session_message usage, billed from the console:

   start (local)	gap since previous request	cached	hit	billed
   09-25 17:42:03	resumed after ~4 days idle (cold start, fine)	384	0.1%	$0.0886
   09-25 17:42:46	43 s	256	0.0%	$0.0889
   09-25 17:43:04	18 s	589,312	100%	$0.0028
   09-25 17:43:20	16 s	586,752	99.2%	$0.0034
   09-26 01:06:01	7.4 h idle (cold start, fine)	384	0.1%	$0.0892
   09-26 01:06:36	35 s	593,024	99.9%	$0.0021
   09-26 01:06:51	15 s	593,536	99.9%	$0.0023
   09-26 01:38:23	31.5 min	256	0.0%	$0.0899

   Two of the four misses are explicable idle cold starts. The other two are the reported pathology: a full ~590k re-read 43 seconds and 31.5 minutes after requests that had just returned ~100% hits on the same session, with a strictly growing prefix. Billing matches the full input rate exactly (590k x $0.15/M ~= $0.0885); the catalog still advertises cache read at $0.003/M, unchanged.

   Token hit rate on my traffic: 99.07% on 09-21 (274 requests) -> 49.94% on 09-25 -> 49.97% on 09-26.

   Why this matters beyond one account: the Go plan advertises $10/month -> ~$60 of DeepSeek Flash usage (~6x). That arithmetic only works with high cache-hit rates — OpenCode's own request estimate assumes ~68k cache-read tokens vs ~790 fresh input per request. When cache-read collapses to ~256 on turn starts, each 590k-context turn costs ~$0.089 instead of ~$0.002 (~45x), so the $60 credit buys roughly an order of magnitude less. In the window above, $0.37 was billed and ~$0.35 of that was pure cache-failure waste.

   This lines up with the two findings already in this thread: the byte-identical raw-HTTP flap (upstream replica/eviction behavior, not client-side), and the source analysis (client sends session affinity headers unconditionally; the gateway does not use them for sticky routing). Nothing client-side can recover credit that has already been billed as full-context input. Fix has to be gateway-side cache affinity (or a shared cache layer).

   👍React with 👍1Reacted by geminigeek
7. neriousy commented on Oct 21, 2026

   

   neriousy

   on Oct 21, 2026

   Member

   More actions

   Thanks for the thorough analysis. This was most likely fixed since it was reported: Go routing and caching for DeepSeek have changed a lot. Closing because there's been no activity for a while. If you run into full-context re-reads again, please open a new issue with the time, session ID and request IDs from the console logs. If it used up your allowance, email help@anoma.ly with your workspace ID.
8. neriousy

   closed this as not plannedon Oct 21, 2026