Parallel tool child cancellation leaves sibling tasks running past invocation completion

Open
#7,172 0 comments 0 reactions 1 assignee View on GitHub

@llalitkumarrr is already working on this.

Since Sep 18, 2026.

Assessment

This issue has not been assessed yet.

Description

Problem

When one tool in a parallel tool batch undergoes child-originated cancellation, the batch can return without cancelling and awaiting its sibling tool calls. Here, child-originated cancellation means either the tool raises asyncio.CancelledError, or it cancels its own task with asyncio.current_task().cancel() and reaches an await. The caller does not cancel the batch.

With the public Runner, the invocation iterator finishes while a sibling tool is still executing. That sibling can subsequently perform a side effect. Callback ordering is an observable symptom; the core issue is that the parallel batch returns while a child task it spawned remains unsettled.

Minimal reproduction

  1. Use a development checkout of google/adk-python at f33d4923388a963d0c5cbf8f7855a88d255b0dea (also the latest main when checked on 2026-09-18), with its Python dependencies installed.
  2. Save the following as /tmp/adk-child-cancel.py.
  3. From the repository root, run PYTHONPATH=src:. python /tmp/adk-child-cancel.py using that environment.

This uses the repository's MockModel to supply one parallel function-call response. It does not use LiteLLM, a model API, or a network service.

import asyncio

from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
from tests.unittests.testing_utils import MockModel


async def main():
    trace = []
    started, release = asyncio.Event(), asyncio.Event()
    sibling_task = None

    async def stopper() -> dict:
        """Cancel this tool after its sibling starts."""
        await started.wait()
        trace.append("stopper:self_cancel")
        asyncio.current_task().cancel()
        await asyncio.sleep(0)  # Deliver this task's own cancellation.
        return {}

    async def sibling() -> dict:
        """Represent a delayed side effect."""
        nonlocal sibling_task
        sibling_task = asyncio.current_task()
        trace.append("sibling:start")
        started.set()
        try:
            await release.wait()
            trace.append("sibling:side_effect")  # Local append only.
            return {"ok": True}
        finally:
            trace.append("sibling:finally")

    async def after_agent(callback_context):
        trace.append("after_agent")

    async def after_tool(tool, args, tool_context, tool_response):
        trace.append("after_tool:" + tool.name)

    model = MockModel.create(responses=[[
        types.Part(function_call=types.FunctionCall(name=n, id=n, args={}))
        for n in ("stopper", "sibling")
    ]])
    agent = LlmAgent(
        name="probe", model=model, tools=[stopper, sibling],
        after_agent_callback=after_agent, after_tool_callback=after_tool,
    )
    service = InMemorySessionService()
    runner = Runner(app_name="scout", agent=agent, session_service=service)
    session = await service.create_session(app_name="scout", user_id="u")
    try:
        async for _ in runner.run_async(
            user_id="u", session_id=session.id,
            new_message=types.Content(
                role="user", parts=[types.Part(text="Run both tools.")]
            ),
        ):
            pass
        trace.append("runner_returned:complete")
        pending = sibling_task is not None and not sibling_task.done()
    finally:
        # Release/drain any surviving sibling so the repro itself leaks no task.
        release.set()
        if sibling_task is not None:
            await asyncio.gather(sibling_task, return_exceptions=True)
        await runner.close()

    print("\n".join(trace))
    print(f"sibling_pending_at_runner_return={pending}")


asyncio.run(main())
Observed output
sibling:start
stopper:self_cancel
after_agent
runner_returned:complete
sibling:side_effect
sibling:finally
after_tool:sibling
sibling_pending_at_runner_return=True

The sibling is gated with asyncio.Event, so this does not depend on timing sleeps. The only sleep(0) delivers the tool task's own cancellation. The local sibling:side_effect trace append stands in for an external write; the reproducer performs no external side effect and drains the surviving task before exiting.

Expected behavior

A parallel tool batch should not return while sibling tasks spawned by that batch are still running. If one child terminates the batch through cancellation, unfinished siblings should be cancelled and awaited before the original cancellation is propagated out of the batch.

The invariant is that sibling cleanup completes before the batch exits. This does not require converting cancellation into a normal tool error, changing callbacks to run from finally blocks, or changing how Runner exposes root-node cancellation.

Reproduction matrix

Condition Async: sibling pending at batch return Live: sibling pending at batch return
Normal completion no no
Tool raises ValueError no no
Caller cancels the entire batch no no
Tool raises CancelledError yes yes
Tool cancels its own task yes yes

Separate full Runner reproductions cover run_async with StreamingMode.NONE, run_async with SSE, and run_live with BIDI. All three exhibit the same child-cancellation behavior. Live uses the repository's MockModel, not a live model service.

Root cause and candidate repair

_gather_or_cancel catches Exception, cancels unfinished child tasks, awaits their cleanup, and re-raises. asyncio.CancelledError derives from BaseException, so child-originated cancellation bypasses that path. asyncio.gather() propagating a child's cancellation does not itself cancel its siblings.

A minimal candidate repair is to include asyncio.CancelledError in the existing cleanup path, for example with except (Exception, asyncio.CancelledError), then reuse the existing cancel/await/re-raise logic. I tested this as a process-local monkeypatch only; production source was not modified. Cancellation remains cancellation, rather than becoming a tool error response.

Verification and environment

Baseline: f33d4923 (2026-09-17), macOS arm64, Python 3.11.9.

  • Existing BaseAgent, async tool callback, and live tool callback suites: 111 passed.
  • Local scout matrix on unmodified main: 21 passed, 10 failed; all failures concern surviving siblings after child-originated cancellation.
  • Same matrix with the process-local candidate repair: 31 passed.
  • Stability: 40/40 deterministic reproductions (10 rounds × 2 tool execution modes × 2 child-cancellation forms).
  • Dependencies: google-genai 2.22.0, pydantic 2.12.5, pytest 9.1.1, pytest-asyncio 1.4.0.

The existing local environment was reused. Full supported-Python-version tests and real-backend live E2E have not been run. I have not established which release first introduced this behavior. I can provide the broader Runner probe and regression matrix if useful, and am happy to send a focused PR if this direction fits.

Duplicate check

I searched current issue/PR listings and cancellation-related terms and did not find the same case. Nearby reports differ:

  • #7058 / #7059: concurrent streaming tools with the same name overwrite task tracking. This repro uses two distinct ordinary async tools.
  • #6559 / #6560: OpenTelemetry context errors during early iterator close. This repro fully consumes the Runner iterator.
  • #4796: public stop-generating support. This report concerns cleanup of sibling tasks already spawned by a parallel tool batch.
Dominant language
Python
Stars
21.6k
Forks
4k
Avg merge
13h 49m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from google/adk-python

All issues in google/adk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.