google / google/adk-python

2.7.0 on Agent Engine: worker dies without a traceback on the second model call after a tool round-trip, truncating the stream

Open
#6,757 5 comments 0 reactions 1 assignee Claimed by @llalitkumarrr View on GitHub
agent engine request clarification
Dominant language
Python
Stars
21.5k
Forks
4k
Avg merge
1d 14h
Merged PRs (30d)
37

Description

## 🔴 Required Information

**Bug description:**

After upgrading `google-adk` 2.6.2 → 2.7.0 (with `google-cloud-aiplatform` 1.163.0 → 1.164.0), every invocation that involves a tool call fails on Vertex AI Agent Engine. The uvicorn worker process handling the request **disappears without emitting a Python traceback**, and the parent process starts a replacement worker a few seconds later. The `:streamQuery` response ends after two events, so the client sees a well-formed HTTP 200 stream that simply contains no model text.

The invocation dies **after** the tool round-trip completes successfully — the failure is in the second model call of the turn.

Reverting only the two dependencies (no application code change) makes the same question complete normally.

**Steps to Reproduce:**

1. Build an `LlmAgent` as the root agent (`coordinator`) with:
- a function tool that performs network I/O (in our case a Workspace search returning ~5 results),
- seven `sub_agents`, all `mode="single_turn"` (five domain specialists, one artifact reader, one grounding judge),
- `before_model_callback` that sets `tool_config` (`FunctionCallingConfigMode.ANY`) on the first LLM call of a turn,
- `after_agent_callback` returning `types.Content` when the root produced no visible text,
- `BigQueryAgentAnalyticsPlugin` registered via `AdkApp(agent=..., plugins=[...])`.
2. Deploy to Agent Engine (`vertexai.agent_engines`, `agent_framework="google-adk"`), region `asia-northeast1`, Python 3.13.
3. Send a question through `:streamQuery` that requires the tool.

**Expected Behavior:**

The turn proceeds: model → tool call → tool result → model → delegation to a sub-agent → final text. This is exactly what 2.6.2 does (14 events, final answer with citations).

**Observed Behavior:**

The stream ends after 2 events:

```jsonc
// event 1 — coordinator, function_call + thought_signature
{"author":"coordinator","content":{"parts":[{"function_call":{"name":"search_workspace_documents", ...},
"thought_signature":"..."}],"role":"model"},"finish_reason":"STOP", ...}

// event 2 — coordinator, function_response (tool succeeded, 5 results)
{"author":"coordinator","content":{"parts":[{"function_response":{"name":"search_workspace_documents",
"response":{"status":"success","results":[ ... 5 items ... ]}}}],"role":"user"}, ...}

// nothing further
```

Runtime logs for the same invocation:

```
03:40:04.415 [26] POST /api/stream_reasoning_engine HTTP/1.1" 200 OK
03:40:06.514 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:07.692 [26] Response received from the model.
(tool executes successfully, ~5.4 s)
03:40:13.100 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:17.518 [413] gcsfs experimental features enabled ... <- different PID: a new worker booted
03:40:17.529 [413] Plugin 'bigquery_agent_analytics' registered.
03:40:17.532 [413] Application startup complete.
```

Worker `[26]` never logs `Response received from the model.` for the second call. **No traceback, no ERROR-level entry** appears anywhere in the invocation window, which points to the process being killed by a signal rather than raising.

**What we ruled out:**

| Hypothesis | Result |
| --- | --- |
| Model generation | Reproduced with both `gemini-2.5-flash` and `gemini-3.5-flash` |
| The tool itself | Tool completed in 5.4 s and returned `status: success` with 5 results |
| Auth / credentials | `status: success`, no `auth_required` |
| Application code | Unchanged across the working and failing deploys; only the two dependencies differ |
| `tool_config` forcing not being released | Verified with a real `Runner` on 2.7.0: the second call has `tool_config=None` as expected |
| Dependency set growth | `uv export --only-group runtime` diff is 3 lines (adk, aiplatform, and `google-cloud-iam` dropped); no new packages |

**Possibly related observation (not a proven cause):**

Container memory allocation for the same question roughly doubled: ~0.8 GiB on 2.6.2 vs ~1.6–2.0 GiB on 2.7.0 (Agent Engine console metric).

We also measured that `LlmAgent._resolved_model` (new in 2.7.0) keeps one resolved `BaseLlm` per agent. With 8 agents that all pass the model as a **string**, this retains 8 distinct `Gemini` instances and 8 distinct `genai.Client` objects (+18.5 MB RSS per worker in our measurement). That alone is far too small to explain a kill, but it does mean 2.7.0 retains state where 2.6.2 let it be collected.

**Environment Details:**

- ADK version: 2.7.0 (working: 2.6.2)
- google-cloud-aiplatform: 1.164.0 (working: 1.163.0)
- Python: 3.13
- Runtime: Vertex AI Agent Engine (`reasoningEngines`), region `asia-northeast1`, 11 uvicorn workers
- Model: `gemini-2.5-flash` / `gemini-3.5-flash`, `stream: False`, Vertex backend
- Telemetry: `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`
- Plugins: `BigQueryAgentAnalyticsPlugin`

Happy to provide the full NDJSON stream, additional log windows, or to run a targeted experiment (e.g. with telemetry disabled) if that would help narrow it down.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.