LiteLlm finalizes a stream cut off mid-answer as finish_reason STOP with no error

🔴 Required Information

Describe the Bug:
When a LiteLlm stream ends without a finish chunk (the connection drops mid-answer), generate_content_async still finalizes the buffered text with finish_reason=STOP and no error_code. The fallback is last_finish_reason or "stop" (lite_llm.py#L4587, #L4594, #L4608). So a truncated answer looks the same as a finished one to the agent, to callbacks and to telemetry. Came up in #7630.

Steps to Reproduce:

pip install google-adk litellm uvicorn starlette (ran it on main, 2de5876)
Run the script below. A local OpenAI-compatible server streams two chunks and then drops the connection for the cut model, no API key needed.

repro.py

import asyncio, json, threading, time

import uvicorn
from starlette.applications import Starlette
from starlette.responses import StreamingResponse
from starlette.routing import Route

from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.genai import types


def chunk(delta, finish=None):
    return {"id": "c1", "object": "chat.completion.chunk", "created": int(time.time()), "model": "gpt-4o",
            "choices": [{"index": 0, "delta": delta, "finish_reason": finish}]}


async def completions(request):
    cut = (await request.json())["model"] == "cut"

    async def sse():
        yield f"data: {json.dumps(chunk({'role': 'assistant', 'content': 'Oslo is '}))}\n\n"
        yield f"data: {json.dumps(chunk({'content': '4C and'}))}\n\n"
        if cut:
            raise ConnectionResetError("server dropped the stream")
        yield f"data: {json.dumps(chunk({'content': ' cloudy.'}))}\n\n"
        yield f"data: {json.dumps(chunk({}, 'stop'))}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(sse(), media_type="text/event-stream")


server = uvicorn.Server(uvicorn.Config(
    Starlette(routes=[Route("/v1/chat/completions", completions, methods=["POST"])]),
    port=8797, log_level="critical"))
threading.Thread(target=server.run, daemon=True).start()
while not server.started:
    time.sleep(0.05)


async def main():
    for model in ("complete", "cut"):
        llm = LiteLlm(model=f"openai/{model}", api_base="http://127.0.0.1:8797/v1", api_key="unused")
        request = LlmRequest(model=f"openai/{model}", contents=[types.Content(role="user", parts=[types.Part(text="weather?")])])
        last = None
        async for response in llm.generate_content_async(request, stream=True):
            last = response
        text = "".join(p.text or "" for p in last.content.parts)
        print(f"{model:9} final: partial={last.partial} finish_reason={last.finish_reason} error_code={last.error_code} text={text!r}")

asyncio.run(main())
server.should_exit = True

Expected Behavior:
A stream that ends without a finish reason is reported as incomplete, for example an error_code or a finish reason other than STOP, so callers and telemetry can tell it was cut off.

Observed Behavior:

complete  final: partial=False finish_reason=FinishReason.STOP error_code=None text='Oslo is 4C and cloudy.'
cut       final: partial=False finish_reason=FinishReason.STOP error_code=None text='Oslo is 4C and'

Environment Details:

ADK Library Version (pip show google-adk): main at 2de5876
Desktop OS: macOS
Python Version (python -V): 3.12

Model Information:

Are you using LiteLLM: Yes, litellm 1.85.7
Which model is being used: openai provider against a local server (same code path for any LiteLlm provider)

🟡 Optional Information

Regression:
Haven't checked older versions.

Not sure which signal you'd want there, an error_code or a different finish reason, so leaving the fix open. Happy to send a PR once that's decided.

@surajksharma07

I dug into this with a chunk-level probe and found the truncation signal is erased inside litellm, before ADK's fallback can matter — so the fix for the live scenario probably can't live at the lines linked above.

What the probe shows. A local OpenAI-compatible server streams two text chunks and then either closes gracefully (no finish chunk, no [DONE]) or drops the connection. Capturing the raw chunks litellm delivers to LiteLlm:

litellm	graceful close (no finish chunk)	hard reset
1.85.7 (the version in the report)	`[('Oslo is ', None), ('4C and', None), (None, 'stop')]` — a synthetic stop chunk	silent end, same synthetic stop (matches the reported behavior exactly)
1.106.2 (current cap)	same synthetic stop	raises `APIConnectionError` (`MidStreamFallbackError`)

So by the time ADK's streaming loop ends, last_finish_reason is already "stop" — the last_finish_reason or "stop" fallbacks never fire in this scenario, and changing them cannot alter the reported outcome. The evidence is gone before ADK sees the stream.

Where I think this leaves it:

The durable fix for the reported behavior belongs upstream in litellm (don't synthesize a finish chunk when the provider never sent one, or expose an incomplete-stream signal).
ADK-side, last_finish_reason is None at finalization is still reachable through custom llm_client implementations that stream raw chunks, and finalizing those as a clean STOP is arguably still wrong — I have a small local patch that maps that case to FinishReason.OTHER + error_code (following _apply_provider_finish_reason's existing convention), with mocked-stream tests. Happy to send it if you want the contract hardened independently of litellm, but I don't want to present it as fixing the live repro.

@surajksharma07 how would you like to proceed — report/track upstream with litellm, harden the ADK contract anyway, or both?

@ishan-1010

Seeing the same on main (2.13.0) with litellm 1.84.0 @ishan-1010. litellm adds a synthetic stop on the cut stream and a raw llm_client stream with no finish reason also ends up as STOP via the or "stop" fallback.

As a workaround you could try a small LiteLLMClient wrapper that checks the stream’s received_finish_reason and rewrites the synthetic stop (or appends a final chunk) as "incomplete" so ADK reports it as FinishReason.OTHER with error_code set. Both cases work that way on my side against a local SSE server.

Could you check if that works with your setup too including a tool-call stream that gets cut off?

Both make sense: harden the ADK fallback to OTHER + error_code like you suggested and raise the synthetic stop with litellm upstream. Please test the patch well with mocked streams before raising the PR.

@surajksharma07

@surajksharma07 the wrapper works here. main (2084efb), litellm 1.85.7, local SSE server, wrapper rewrites the finish reason when received_finish_reason is still None:

stream	default `LiteLLMClient`	wrapper
text or tool call, completes normally	STOP	STOP
text, server closes without a finish chunk	STOP, no `error_code`	OTHER + `error_code`
text, connection reset	STOP, no `error_code`	OTHER + `error_code`
tool call with complete args, then close or reset	STOP, no `error_code`	OTHER + `error_code`
tool call cut mid-args	MALFORMED_FUNCTION_CALL	same

one thing for the hardened fallback: a tool call cut after its args arrived still comes back with the call in its parts, only error_code marks it incomplete. should that call still be offered to the flow?

wrapper and test script

"""Check the LiteLLMClient wrapper workaround from #7693 against cut-off text and tool-call streams."""
import asyncio, json, threading, time

import uvicorn
from starlette.applications import Starlette
from starlette.responses import StreamingResponse
from starlette.routing import Route

from google.adk.models.lite_llm import LiteLlm, LiteLLMClient
from google.adk.models.llm_request import LlmRequest
from google.genai import types


def chunk(delta, finish=None):
    return {"id": "c1", "object": "chat.completion.chunk", "created": int(time.time()), "model": "gpt-4o",
            "choices": [{"index": 0, "delta": delta, "finish_reason": finish}]}


async def completions(request):
    kind, ending = (await request.json())["model"].split("-")

    async def sse():
        if kind == "text":
            yield f"data: {json.dumps(chunk({'role': 'assistant', 'content': 'Oslo is '}))}\n\n"
            yield f"data: {json.dumps(chunk({'content': '4C and'}))}\n\n"
        else:
            yield f"data: {json.dumps(chunk({'role': 'assistant', 'tool_calls': [{'index': 0, 'id': 'call_1', 'type': 'function', 'function': {'name': 'get_weather', 'arguments': ''}}]}))}\n\n"
            yield f"data: {json.dumps(chunk({'tool_calls': [{'index': 0, 'function': {'arguments': '{\"city\": \"Os'}}]}))}\n\n"
            if ending != "cutargs":
                yield f"data: {json.dumps(chunk({'tool_calls': [{'index': 0, 'function': {'arguments': 'lo\"}'}}]}))}\n\n"
        if ending in ("reset", "cutargs"):
            raise ConnectionResetError("server dropped the stream")
        if ending == "complete":
            yield f"data: {json.dumps(chunk({}, 'stop' if kind == 'text' else 'tool_calls'))}\n\n"
            yield "data: [DONE]\n\n"
        # ending == "close": the stream just ends, no finish chunk and no [DONE]

    return StreamingResponse(sse(), media_type="text/event-stream")


class CutOffAwareClient(LiteLLMClient):
  """Reports a stream whose provider never sent a finish reason as incomplete."""

  async def acompletion(self, model, messages, tools, **kwargs):
    stream = await super().acompletion(model=model, messages=messages, tools=tools, **kwargs)
    if not kwargs.get("stream"):
      return stream

    async def checked():
      async for part in stream:
        for choice in getattr(part, "choices", None) or []:
          if choice.finish_reason and stream.received_finish_reason is None:
            choice.finish_reason = "incomplete"
        yield part

    return checked()


server = uvicorn.Server(uvicorn.Config(
    Starlette(routes=[Route("/v1/chat/completions", completions, methods=["POST"])]),
    port=8796, log_level="critical"))
threading.Thread(target=server.run, daemon=True).start()
while not server.started:
    time.sleep(0.05)


async def run(name, client):
    llm = LiteLlm(model=f"openai/{name}", api_base="http://127.0.0.1:8796/v1", api_key="unused", llm_client=client)
    request = LlmRequest(model=f"openai/{name}", contents=[types.Content(role="user", parts=[types.Part(text="weather?")])])
    last, error = None, None
    try:
        async for response in llm.generate_content_async(request, stream=True):
            last = response
    except Exception as e:  # noqa: BLE001
        error = type(e).__name__
    if last is None:
        return f"raised {error}"
    parts = last.content.parts if last.content else []
    shown = [p.text if p.text else f"call({p.function_call.name}, {p.function_call.args})" for p in parts if p.text or p.function_call]
    return f"finish={last.finish_reason.name if last.finish_reason else None} error_code={last.error_code} parts={shown}"


async def main():
    for name in ("text-complete", "text-close", "text-reset", "tool-complete", "tool-close", "tool-reset", "tool-cutargs"):
        print(f"{name:14} default : {await run(name, LiteLLMClient())}")
        print(f"{name:14} wrapper : {await run(name, CutOffAwareClient())}")

asyncio.run(main())
server.should_exit = True

text-complete  default : finish=STOP error_code=None parts=['Oslo is 4C and']
text-complete  wrapper : finish=STOP error_code=None parts=['Oslo is 4C and']
text-close     default : finish=STOP error_code=None parts=['Oslo is 4C and']
text-close     wrapper : finish=OTHER error_code=FinishReason.OTHER parts=['Oslo is 4C and']
text-reset     default : finish=STOP error_code=None parts=['Oslo is 4C and']
text-reset     wrapper : finish=OTHER error_code=FinishReason.OTHER parts=['Oslo is 4C and']
tool-complete  default : finish=STOP error_code=None parts=["call(get_weather, {'city': 'Oslo'})"]
tool-complete  wrapper : finish=STOP error_code=None parts=["call(get_weather, {'city': 'Oslo'})"]
tool-close     default : finish=STOP error_code=None parts=["call(get_weather, {'city': 'Oslo'})"]
tool-close     wrapper : finish=OTHER error_code=FinishReason.OTHER parts=["call(get_weather, {'city': 'Oslo'})"]
tool-reset     default : finish=STOP error_code=None parts=["call(get_weather, {'city': 'Oslo'})"]
tool-reset     wrapper : finish=OTHER error_code=FinishReason.OTHER parts=["call(get_weather, {'city': 'Oslo'})"]
tool-cutargs   default : finish=MALFORMED_FUNCTION_CALL error_code=MALFORMED_FUNCTION_CALL parts=[]
tool-cutargs   wrapper : finish=MALFORMED_FUNCTION_CALL error_code=MALFORMED_FUNCTION_CALL parts=[]

@ishan-1010

That table lines up with what I saw @ishan-1010 and good that cutargs still lands on MALFORMED_FUNCTION_CALL either way.

On your question: today it does get offered. The flow only skips partial events before dispatching function calls and never looks at error_code (base_llm_flow.py:507-521) so with the wrapper the tool-close/tool-reset calls still run. #7741 (ef76578) goes the same way on purpose. It only marks the text-only finalizer as truncated and keeps or "tool_calls" treating args that fully parsed as the completeness check. That’s right: a parsed call is complete and the only real loss is a later parallel call that never arrived. If maintainers want stricter the gate belongs in the flow not in lite_llm.

Two notes on #7741: it covers the raw-client fallback but not your live case since litellm has already set "stop" there so a received_finish_reason check like your wrapper’s would be a natural follow-up. @xxwtiancai could you also add a test pinning that a tool-call stream with no finish chunk stays STOP and run the full test_litellm.py before review?

@ishan-1010 would you be up for filing the synthetic-stop case with litellm using your probe? Then we can link it here.

@surajksharma07

@surajksharma07 it's already filed upstream as BerriAI/litellm#46542 (open since Oct 14, same finish_reason_handler fallback to "stop"). still reproduces on 1.106.4 with a clean close. a connection reset does raise MidStreamFallbackError there, so only the clean close is hidden. two fix PRs are open for it: BerriAI/litellm#40266 (raise on a clean EOF) and BerriAI/litellm#42140 (opt-in strict mode). linking it here rather than filing a duplicate.

@surajksharma07

@surajksharma07 one correction, LiteLLM staff have a fix for this in BerriAI/litellm#47665 (approved, not merged yet). a stream that ends without a finish reason will raise instead of returning stop.

@ishan-1010

Thanks for tracking down BerriAI/litellm#46542 and the #47665 fix @ishan-1010, much better than filing a duplicate. One catch: ADK currently pins litellm>=1.84,<1.106.3 so #47665 won’t reach ADK users until that pin is raised after the release is checked.

Once it is a clean close should raise out of the stream loop (lite_llm.py has no try around acompletion at ~L5007) so the run errors the same way a reset already does on 1.106.x instead of ending as a truncated STOP. Your received_finish_reason wrapper is still a good workaround until then. Could you re-run your probe on the litellm release that ships #47665 and confirm it raises for both text and tool-call streams?

#7741 is still needed for raw llm_client streams that litellm never touches. @xxwtiancai the tool-call case is pinned by test_streaming_tool_call_cut_off_after_complete_arguments_is_kept and the full test_litellm.py passes (770) which covers what I asked for. Please keep testing it against main before review.

Will flag #7741 for maintainer review and mention the litellm pin here so the bump can be tracked once #47665 is released.

Sure, will do once it's released. BerriAI/litellm#47665 isn't merged yet.

@ishan-1010

Quick update @ishan-1010: BerriAI/litellm#47665 was merged on Oct 31 (efc2304) but it isn’t in a release yet. 1.106.6 is the latest stable and 1.108.0rc1 / 1.109.0.dev2 were cut before the merge. It’s tagged as a breaking change and raises MidStreamFallbackError (“Stream ended before a finish_reason was received”) while empty streams and mock_response still end in stop so ADK’s mock-based tests shouldn’t change.

On the ADK side main (c3abc93) still has the or "stop" fallbacks (lite_llm.py ~L5276-5297) and the litellm<1.106.3 pin so your wrapper is still the workaround for now. #7815 only changed telemetry so it doesn’t touch this.

#7741 still merges cleanly on current main but hasn’t had a review yet. @xuanyang15 could you take a look when you get a chance? It covers the raw llm_client path which the litellm fix won’t reach.

When a litellm release with #47665 ships please re-run your probe for text and tool-call streams so we can check the run errors out cleanly before the pin is raised.

fix: report a LiteLlm stream without a finish reason as truncated

Navigation Menu

Repository navigation

LiteLlm finalizes a stream cut off mid-answer as finish_reason STOP with no error #7693

🔴 Required Information

🟡 Optional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

LiteLlm finalizes a stream cut off mid-answer as finish_reason STOP with no error #7693

Description

🔴 Required Information

🟡 Optional Information

Activity

xxwtiancai commented on Oct 23, 2026

surajksharma07 commented on Oct 26, 2026

ishan-1010 commented on Oct 27, 2026

surajksharma07 commented on Oct 27, 2026

ishan-1010 commented on Oct 27, 2026

ishan-1010 commented on Oct 28, 2026

surajksharma07 commented on Oct 28, 2026

ishan-1010 commented on Oct 29, 2026

surajksharma07 commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions