Gemini API latency has recently increased significantly, especially with store=true #3059

Activity

1. GTing31

   added

   type: questionRequest for information or clarification. Not an issue.Request for information or clarification. Not an issue.

   priority: p3Desirable enhancement or fix. May not be included in next release.Desirable enhancement or fix. May not be included in next release.

   on Oct 6, 2026
2. Venkaiahbabuneelam commented on Oct 6, 2026

   

   Venkaiahbabuneelam

   on Oct 6, 2026

   More actions

   Hi @GTing31, thanks for all the details! Could you share the exact code snippet of your receive loop? We'd love to try reproducing this on our end using the exact logic you have set up.
   Thanks
3. Venkaiahbabuneelam

   added

   status:awaiting user response

   on Oct 6, 2026
4. Venkaiahbabuneelam

   self-assigned this

   on Oct 6, 2026
5. GTing31 commented on Oct 7, 2026

   

   GTing31

   on Oct 7, 2026

   Author

   More actions

   Hi, thanks for looking into this!

   Below is a minimal example preserving the receive-loop logic from our hello test. We use LiteLLM's Interactions interface; the current environment has LiteLLM 1.96.2.

   Timing starts before the API call. We record time to first text when the first non-empty delta.text arrives, and continue consuming the stream through interaction.completed and stream exhaustion. There is no early exit or application-level retry in this example.

   Set GEMINI_API_KEY in the environment before running:

   import asyncio
   import os
   import time
   
   import litellm
   
   
   async def measure(store):
       started = time.perf_counter()
       first_text = None
       completion = None
       pieces = []
   
       stream = await litellm.interactions.acreate(
           model="gemini/gemini-3.5-flash-lite",
           input="只說hello",
           generation_config={
               "thinking_level": "minimal",
               "temperature": 0.0,
               "max_output_tokens": 32,
           },
           store=store,
           stream=True,
           api_key=os.environ["GEMINI_API_KEY"],
           timeout=60,
       )
   
       async for event in stream:
           data = event.model_dump()
           elapsed = time.perf_counter() - started
           event_type = data.get("event_type")
           part = (data.get("delta") or {}).get("text")
   
           if isinstance(part, str) and part:
               first_text = elapsed if first_text is None else first_text
               pieces.append(part)
   
           if event_type == "interaction.completed":
               completion = elapsed
   
       return {
           "store": store,
           "first_text_s": first_text,
           "completed_s": completion,
           "total_s": time.perf_counter() - started,
           "answer": "".join(pieces),
       }
   
   
   async def main():
       for store in (True, False):
           print(await measure(store))
           await asyncio.sleep(1)
   
   
   asyncio.run(main())

   Our October 6 hello test had 10 requests per setting:

   Median latency	store=true	store=false
   Time to first text	3.340 s	0.625 s
   Full stream duration	4.993 s	0.683 s

   All 20 requests succeeded. This example runs one request per setting; the reported medians are from the earlier 20-request batch.

   Separately, direct REST/curl requests also showed the storage-related delay. Those calls bypassed LiteLLM and the Python SDK.
   To reiterate, our main concern is the significant increase in latency recently. With store=true, Gemini 3.5 Flash-Lite’s median time to first text increased from approximately 1.2 seconds on September 7 to 4.0 seconds on October 6 — about 2.8 seconds of additional waiting before any text appears.

   We also observed an increase with Gemini 3.7 Flash, from approximately 1.3 seconds to 5.1 seconds. Could your team investigate whether any recent backend changes might explain this?

   thanks for your help!
6. Venkaiahbabuneelam commented on Oct 12, 2026

   

   Venkaiahbabuneelam

   on Oct 12, 2026

   More actions

   Hi @GTing31, Thank you for sharing the reproduction snippet! We were able to reproduce the latency difference between store=True and store=False on our end using the native google-genai SDK. we will file a bug internally.
   Thanks
7. Venkaiahbabuneelam

   removed

   status:awaiting user response

   on Oct 12, 2026
8. GTing31 commented on Oct 30, 2026

   

   GTing31

   on Oct 30, 2026

   Author

   More actions

   Hi @Venkaiahbabuneelam,

   I'm following up on your October 12 update, where you confirmed reproducing the latency difference between store=True and store=False using the native google-genai SDK and mentioned filing an internal bug.

   Could you please share any updates on the investigation? Has the internal bug been filed, and has the team identified the cause of the increased latency?

   As described in the original report, the main concern is the additional delay before the first text appears, which significantly affects interactive use when conversation storage is enabled.

   If a fix is planned, is there an estimated timeline? In the meantime, are there any recommended workarounds to reduce this delay while keeping store=True?

   Please let me know if additional measurements or reproduction details would help.

   Thank you for your help!