/run_sse stream stays open when a ParallelAgent branch has no terminal event

[dlwldnjs1009]

Description

With google-adk==1.31.1 + LiteLlm in SSE streaming mode, one leaf of a
ParallelAgent can stop emitting events after tool responses have been
returned, without emitting a final event (partial=false, finishReason=STOP)
and without any exception appearing in the pod stdout or being yielded on the
SSE stream as an error event.

When that happens the parent ParallelAgent never completes, the enclosing
SequentialAgent never advances to the next step, and the /run_sse HTTP
stream remains open — no further data: event, no EOF. The downstream SSE
consumer eventually gives up at its own timeout layer.

We have observed this in two invocations in production traffic today; both
stall in the same leaf agent (sub_workout_recommend_agent) after it has
completed a tool-call round and started streaming its text response. Sibling
branches in the same ParallelAgent complete normally.

Environment

• google-adk: 1.31.1
• litellm: 1.83.12
• Python: 3.13.11
• Container OS: Debian GNU/Linux 13 (trixie)
• Host kernel: Linux 6.8.0
• Package location: /usr/local/lib/python3.13/site-packages
• Runtime: Kubernetes pod, FastAPI app produced by
  AdkWebServer.get_fast_api_app(), endpoint /run_sse
• vLLM (server-side): 0.19.1.dev6+g6d4a8e6d2 (git dev build from commit 6d4a8e6d2, not a tagged release)
• Upstream model served by vLLM: google/gemma-4-31B-it (max_model_len=26000), exposed as alias ait-edge-da
• LiteLlm model string: hosted_vllm/ait-edge-da
• LiteLlm additional args:

  model = LiteLlm(model="hosted_vllm/ait-edge-da", api_base="<internal-url>")
  model._additional_args["extra_body"] = {
      "chat_template_kwargs": {"enable_thinking": False}
  }

• Agent structure:
  • SequentialAgent
    • LlmAgent (health status analysis, text only)
    • ParallelAgent
      • LlmAgent sub_workout_recommend_agent (3 tools)
      • LlmAgent sub_sleep_recommend_agent (1 tool)
    • LlmAgent (summary)
• Frequency: intermittent — 4 occurrences in ~3h20m of production traffic today.

Observed behavior — two stalled invocations

Redacted full event dumps are attached. Values (text chunks, state-delta
fields, tool args, tool response content) are replaced with
[REDACTED_TEXT len=N] / [REDACTED]; structural fields (partial,
finishReason, author, timestamp, id, invocationId, functionCall.name,
functionResponse.name) are kept verbatim.

Invocation A (e-beb6c3e3) — attached: redacted_e-beb6c3e3.jsonl (765 events)

All four partial=false events:

time (KST)	author	role
16:07:34	sub_health_analysis_agent	text final STOP
16:07:35	sub_sleep_recommend_agent	tool-call round STOP
16:07:35	sub_workout_recommend_agent	tool-call round STOP
16:07:56.119	sub_sleep_recommend_agent	sleep text final STOP — last event of invocation

• sub_workout_recommend_agent has no partial=false for its text response.
• sub_summary_agent is never invoked (0 events).
• No events are generated after 16:07:56.119. No EOF on the SSE stream.
• Downstream SSE consumer gives up at 16:12:56
  (= 16:07:56 + 300 s idle timeout).

Invocation B (e-d9e02105) — attached: redacted_e-d9e02105.jsonl (1359 events)

time (KST)	author	role
16:12:55	sub_health_analysis_agent	text final STOP
16:12:55	sub_workout_recommend_agent	tool-call round STOP (3 calls)
16:12:55	sub_sleep_recommend_agent	tool-call round STOP (1 call)
16:13:10	sub_sleep_recommend_agent	sleep text final STOP
16:13:20.318	sub_workout_recommend_agent	last event: partial=true

• sub_workout_recommend_agent has no partial=false for its text response.
• sub_summary_agent is never invoked (0 events).
• No events after 16:13:20.318. Downstream consumer gives up at 16:18:20.

Common pattern

In both invocations:

1. All four tool responses return successfully.
2. sub_sleep_recommend_agent completes its text response normally.
3. sub_workout_recommend_agent does not emit a partial=false for its
   text response. It either stops mid-chunk (B) or produces no text events
   that terminate (A).
4. Because one branch of recommend_parallel_agent never completes, the
   parent SequentialAgent cannot advance to sub_summary_agent.
5. /run_sse stays open without further events and without EOF. Our gateway
   eventually emits its own application-level event after a 300 s idle:

   {"chunkCount":N,"error":"upstream-timeout"}
   

   This payload is from our gateway, not from ADK.

No exception was logged in pod stdout and no error event was yielded on the
SSE stream during the idle window.

Expected behavior

ADK should not leave a branch indistinguishable from an active run indefinitely.
When the upstream model stream ends, errors, or stalls beyond a configured
timeout, the branch should:

• emit a final event (partial=false with finishReason) and let the parent
  agent complete, or
• propagate an exception / error / cancellation event so the parent agent and
  the enclosing workflow can observe and react.

In the observations above there is no final event, no error, and no
cancellation — the branch remains open without an observable terminal signal, which is what makes the stall
undetectable to the parent workflow.

Related

• #5342 — LiteLlm streaming bypass for function-call argument delta
  buffering. Different symptom; does not cover stopped text streaming after
  tool responses.
• #3665 — LiteLlm streaming finish_reason missing (closed). Different
  symptom; the sibling branch in our observation terminates normally with
  finishReason=STOP.

Additional context

• Locally (different versions — litellm 1.83.7, Python 3.14.4) we have
  not reproduced this. The issue may be version- or environment-sensitive,
  but we have not verified which component is the cause.
• Attached redacted traces:
  • redacted_e-beb6c3e3.jsonl — stall A
  • redacted_e-d9e02105.jsonl — stall B
  • redacted_e-7b6bbdbb.jsonl — a successful invocation for comparison
  • was_upstream_timeouts.log — downstream gateway timeouts
  • summary_agent_check.txt — sub_summary_agent event counts per invocation
• LiteLLM DEBUG traces for the same window can be provided on request;
  they contain token-level deltas that likely need additional redaction
  before public posting.

adk-stall-dumps.zip

[dlwldnjs1009]

I ran local probes against the ADK source to validate the mechanism.

What I confirmed:

• If the LiteLlm async stream reaches EOF, ADK synthesizes a final partial=false STOP response even when finish_reason is missing.
• If the upstream async iterator hangs after a partial text chunk, ADK does not emit a final/error event before the iterator resumes, ends, or raises.
• In that state, the ParallelAgent child branch remains open, so the parent SequentialAgent does not advance to the next step.

So this no longer looks like ADK simply dropping a final event after EOF. It looks more like the upstream LiteLLM/vLLM stream, or the transport path feeding it, remains open without EOF/error after the last partial text chunk.

This is still a mechanism check, not a full production root-cause confirmation. The remaining useful evidence would be a raw upstream LiteLLM/vLLM stream trace or task stack captured while the Kubernetes run is stuck.

[dlwldnjs1009]

Follow-up with a production repro captured via a SIGUSR1 dump + a coroutine-chain walker that crosses async-generator boundaries.

Deepest observed wait for the stall task in today's repro:

parallel_agent.py:62   process_an_agent   (async for event in events_for_one_agent)
  -> base_agent.py:297     run_async
  -> llm_agent.py:487      _run_async_impl
  -> base_llm_flow.py:{379, 804, 881, 1239, 1261, 1322}
  -> lite_llm.py:2321      generate_content_async
  -> litellm/.../streaming_handler.py:2017   __anext__
  -> litellm/.../base_model_iterator.py:167  __anext__
  -> httpx/_models.py:{997, 1018, 1031, 1055}  aiter_raw/bytes/text/lines
  -> httpx/_client.py:176  __aiter__
  -> litellm/llms/custom_httpx/aiohttp_transport.py:86  __aiter__
  -> aiohttp/streams.py:{53, 456, 372}  (_wait)   <- chain end at FutureIter

At that moment the pod had no matching outbound TCP socket to the vLLM endpoint and the asyncio loop was idle in ep_poll. The aiohttp StreamReader future is pending and does not resolve.

Our app was not passing timeout into LiteLlm(...), so LiteLLM's aiohttp transport was building ClientTimeout(sock_read=None) — no read-idle escape hatch at the transport layer. We are mitigating on our side by passing timeout=httpx.Timeout(read=60), which the transport maps to ClientTimeout(sock_read=60).

The original observation in this issue stands independently of that fix: even after we set a transport-side timeout, a stuck upstream iterator leaves ParallelAgent and /run_sse pending with no visible error. An opt-in stream-idle timeout on LlmAgent that cancels the leaf and surfaces a terminal error event would cover this regardless of the upstream cause.

Happy to draft a PR adding an optional stream_idle_timeout on LlmAgent. Let me know if Runner or a plugin hook would be the preferred scope instead.

[dlwldnjs1009]

Update on the mitigation angle.

Setting DISABLE_AIOHTTP_TRANSPORT=True changes the failure from a silent stall to an immediate httpx.RemoteProtocolError (malformed chunk footer from the upstream). /run_sse closes fast instead of hanging for 300s.

That shifts but does not close this issue. The ParallelAgent leaf now exits via exception, SequentialAgent never reaches summary, and the generator closes with one error event. A transport-level exception is a better failure mode than a silent hang, but the leaf still has no proper terminal event.

An opt-in stream-idle timeout on LlmAgent that surfaces a well-formed terminal event would cover both the original silent-stall case and this fast-fail case consistently. Still on for the PR — LlmAgent vs Runner, whichever you prefer.

[dlwldnjs1009]

Root cause update from simultaneous packet captures.

Captured the same response on both sides at the same event:

• H100 server egress (where vLLM runs, in a docker container)
• NKS pod receive side (inside the pod's network namespace, via kubectl debug ephemeral container + tcpdump)

Across a 4-minute window with 6 malformed chunk footer events:

• Same 24 HTTP responses on both sides, same body size (1,436,524 bytes total, 10 TCP flows)
• 4 flows: byte-identical
• 6 flows: body length identical but a segment-sized region (40–46 bytes) differs — in-place substitution, not insertion/deletion
• The differing region starts at the same offset inside each affected TCP segment (byte 186) and extends to the segment's end
• Each affected segment carries exactly one chunked HTTP frame (size line + data + footer, ~228–232 bytes)
• The differing bytes on the receive side are random-looking; TCP and IP checksums pass on both sides, so something downstream is recomputing the checksum after substitution

So the malformed chunk footer bytes that h11 reports are real — the bytes actually arrive that way at the client. They are not produced by vLLM/uvicorn/LiteLLM/h11. The corruption is in the network path between the H100 host egress and the NKS pod (in our case a cloud network fabric on the pod side — we've opened a support inquiry with the provider).

This narrows the application-side scope of this issue. I'll keep it open because the original ask — an opt-in terminal/error event from LlmAgent when the upstream stream dies abruptly — is still useful regardless of why the stream died. Happy to close if you'd prefer this be a separate FR.