[Bug][2.5.0 Regression] Workflow replay mixes prior invocation events and breaks same-session HITL resume #6497

Note before submitting: this is a direct reproduction and expansion of
Discussion #6464.
Prefer commenting there or asking a maintainer to convert it into an Issue,
rather than creating a duplicate Issue. If a new Issue is requested, use the
following content.

Describe the Bug

google-adk==2.5.0 regresses Workflow replay for an interruptible workflow
that contains an LlmAgent.

If an ADK session first completes an independent invocation and then starts a
second invocation that pauses on RequestInput, resuming the second invocation
can fail with:

RuntimeError: Replay divergence detected: Timed out waiting for sequence key
'prepare_intent_text@1' to be unblocked.

The failing key belongs to the already completed earlier invocation, not to
the currently resumed HITL invocation.

This makes normal same-session HITL usage unreliable. A session must support
multiple independent user requests, including a later request that requires
human input; requiring a new session for every HITL request is not a viable
application-level workaround.

Steps to Reproduce

1. Install google-adk==2.5.0 with its database/A2A dependencies and configure
   a working LiteLLM OpenAI-compatible endpoint.
2. Use DatabaseSessionService backed by PostgreSQL and create one session.
3. Run the minimal reproduction below with "who are you"; let that first
   invocation complete.
4. Run the same Workflow again in the same session with "weather please".
   It emits a RequestInput event.
5. Resume that exact pending function call with "sample-location".
6. Observe the following error after approximately 15 seconds:

RuntimeError: Replay divergence detected: Timed out waiting for sequence key
'prepare_intent_text@1' to be unblocked.

Expected Behavior

The second invocation should resume the pending clarify node and emit:

resumed:sample-location

Previous completed invocations in the same session must not prevent a later
HITL invocation from resuming.

Observed Behavior

The resumed invocation fails rather than executing the pending HITL node. The
failure points to prepare_intent_text@1, which belongs to the earlier,
already-completed invocation, not the current HITL invocation.

RuntimeError: Replay divergence detected: Timed out waiting for sequence key
'prepare_intent_text@1' to be unblocked.

The exception originates from
google.adk.workflow.utils._replay_sequence_barrier.

Environment Details

ADK Library Version: 2.5.0
Desktop OS: macOS
Python Version: 3.12
Session storage: DatabaseSessionService (PostgreSQL / asyncpg)

Model Information

Using LiteLLM: Yes
Model: glm-5.2 through an OpenAI-compatible LiteLLM gateway

The issue is independent of the model response itself. The failure is raised
from google.adk.workflow.utils._replay_sequence_barrier after its 15-second
wait timeout.

Regression

google-adk==2.4.0: PASS
google-adk==2.5.0: FAIL
google-adk==2.5.0 with a fresh session that directly enters HITL: PASS

Minimal Reproduction Code

This graph deliberately has no graph cycle. clarify is
rerun_on_resume=True, and its interrupt ID includes ctx.run_id so it is
unique for the node execution.

from google.adk import Context, Event, Workflow
from google.adk.agents import LlmAgent
from google.adk.events import RequestInput
from google.adk.workflow import START, node


@node
async def prepare_intent_text(node_input: str, ctx: Context) -> Event:
    return Event(output=node_input)


classifier = LlmAgent(
    name="classifier",
    model=model,  # any working LiteLlm/OpenAI-compatible model
    instruction="Output META for 'who are you'; otherwise output WEATHER.",
    output_key="classification",
)


@node
async def route(node_input: str, ctx: Context) -> Event:
    route_name = "META" if ctx.state.get("classification", "").strip() == "META" else "NEED_INPUT"
    return Event(output=node_input, route=route_name)


@node
async def finish(node_input: str, ctx: Context) -> Event:
    return Event(output="done", message="done")


@node(rerun_on_resume=True)
async def clarify(node_input: str, ctx: Context):
    interrupt_id = f"clarify:{ctx.run_id}"
    response = ctx.resume_inputs.get(interrupt_id)
    if response is None:
        yield RequestInput(interrupt_id=interrupt_id, message="Enter a city")
        return
    yield Event(output=f"resumed:{response}", message=f"resumed:{response}")


workflow = Workflow(
    name="replay_llm_probe",
    edges=[
        (START, prepare_intent_text),
        (prepare_intent_text, classifier),
        (classifier, route),
        (route, {"META": finish, "NEED_INPUT": clarify}),
    ],
)

Against one persistent session, execute:

1. Send "who are you"; allow the invocation to complete.
2. Send "weather please"; the workflow emits RequestInput.
3. Resume the exact pending function call with "sample-location".

Additional Context

The same result occurs after all of the following application-level changes:

• remove the original clarify -> prepare_intent_text graph back-edge;
• set clarify to rerun_on_resume=True;
• create a unique interrupt ID using ctx.run_id (the workaround discussed in RequestInput with constant interrupt_id breaks dev-ui input prompt on @node(rerun_on_resume=True) loops #5617);
• reduce the Workflow to one LlmAgent plus simple FunctionNodes.

Control experiments

This isolates the regression from the application's original loop topology,
the interrupt ID, and model-provider failure.

Suspected cause

The behavior matches cross-invocation contamination of Workflow replay state.
Node path keys such as prepare_intent_text@1 are reused in separate
invocations. If the replay sequence barrier scans terminal events from the
whole session instead of only the current invocation_id, it can wait for an
old key that is never advanced during the current resume.

This is consistent with the rehydration-index optimization introduced around
commit 64a7448fc6e05f37e6dedadd6c0e65bc19264057.

Requested fix

Please ensure all replay sequence construction and event indexing are scoped by
the current invocation ID (or key replay state by both invocation ID and node
path). Please add a regression test that:

1. runs an LlmAgent-containing Workflow to completion;
2. starts another invocation of the same Workflow in the same persistent
   session;
3. pauses with RequestInput and resumes it successfully.

Related reports

• RequestInput with constant interrupt_id breaks dev-ui input prompt on @node(rerun_on_resume=True) loops #5617 documents a separate Dev UI issue involving repeated RequestInput IDs;
  this reproduction uses a unique ID and still fails.
• Discussion [Bug][2.5.0 Regression] Workflow replay mixes events across invocation_ids and breaks consecutive HITL runs #6464 describes the same 2.5.0 cross-invocation replay regression.

How often has this issue occurred?

Always (100%) for the minimal reproduction above.