2.7.0 on Agent Engine: worker dies without a traceback on the second model call after a tool round-trip, truncating the stream
- Vorherrschende Sprache
- Python
- Sterne
- 21.5k
- Forks
- 4k
- Ø Merge
- 1 T. 14 Std.
- Gemergte PRs (30 T.)
- 37
Beschreibung
## 🔴 Required Information
**Bug description:**
After upgrading `google-adk` 2.6.2 → 2.7.0 (with `google-cloud-aiplatform` 1.163.0 → 1.164.0), every invocation that involves a tool call fails on Vertex AI Agent Engine. The uvicorn worker process handling the request **disappears without emitting a Python traceback**, and the parent process starts a replacement worker a few seconds later. The `:streamQuery` response ends after two events, so the client sees a well-formed HTTP 200 stream that simply contains no model text.
The invocation dies **after** the tool round-trip completes successfully — the failure is in the second model call of the turn.
Reverting only the two dependencies (no application code change) makes the same question complete normally.
**Steps to Reproduce:**
1. Build an `LlmAgent` as the root agent (`coordinator`) with:
- a function tool that performs network I/O (in our case a Workspace search returning ~5 results),
- seven `sub_agents`, all `mode="single_turn"` (five domain specialists, one artifact reader, one grounding judge),
- `before_model_callback` that sets `tool_config` (`FunctionCallingConfigMode.ANY`) on the first LLM call of a turn,
- `after_agent_callback` returning `types.Content` when the root produced no visible text,
- `BigQueryAgentAnalyticsPlugin` registered via `AdkApp(agent=..., plugins=[...])`.
2. Deploy to Agent Engine (`vertexai.agent_engines`, `agent_framework="google-adk"`), region `asia-northeast1`, Python 3.13.
3. Send a question through `:streamQuery` that requires the tool.
**Expected Behavior:**
The turn proceeds: model → tool call → tool result → model → delegation to a sub-agent → final text. This is exactly what 2.6.2 does (14 events, final answer with citations).
**Observed Behavior:**
The stream ends after 2 events:
```jsonc
// event 1 — coordinator, function_call + thought_signature
{"author":"coordinator","content":{"parts":[{"function_call":{"name":"search_workspace_documents", ...},
"thought_signature":"..."}],"role":"model"},"finish_reason":"STOP", ...}
// event 2 — coordinator, function_response (tool succeeded, 5 results)
{"author":"coordinator","content":{"parts":[{"function_response":{"name":"search_workspace_documents",
"response":{"status":"success","results":[ ... 5 items ... ]}}}],"role":"user"}, ...}
// nothing further
```
Runtime logs for the same invocation:
```
03:40:04.415 [26] POST /api/stream_reasoning_engine HTTP/1.1" 200 OK
03:40:06.514 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:07.692 [26] Response received from the model.
(tool executes successfully, ~5.4 s)
03:40:13.100 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:17.518 [413] gcsfs experimental features enabled ... <- different PID: a new worker booted
03:40:17.529 [413] Plugin 'bigquery_agent_analytics' registered.
03:40:17.532 [413] Application startup complete.
```
Worker `[26]` never logs `Response received from the model.` for the second call. **No traceback, no ERROR-level entry** appears anywhere in the invocation window, which points to the process being killed by a signal rather than raising.
**What we ruled out:**
| Hypothesis | Result |
| --- | --- |
| Model generation | Reproduced with both `gemini-2.5-flash` and `gemini-3.5-flash` |
| The tool itself | Tool completed in 5.4 s and returned `status: success` with 5 results |
| Auth / credentials | `status: success`, no `auth_required` |
| Application code | Unchanged across the working and failing deploys; only the two dependencies differ |
| `tool_config` forcing not being released | Verified with a real `Runner` on 2.7.0: the second call has `tool_config=None` as expected |
| Dependency set growth | `uv export --only-group runtime` diff is 3 lines (adk, aiplatform, and `google-cloud-iam` dropped); no new packages |
**Possibly related observation (not a proven cause):**
Container memory allocation for the same question roughly doubled: ~0.8 GiB on 2.6.2 vs ~1.6–2.0 GiB on 2.7.0 (Agent Engine console metric).
We also measured that `LlmAgent._resolved_model` (new in 2.7.0) keeps one resolved `BaseLlm` per agent. With 8 agents that all pass the model as a **string**, this retains 8 distinct `Gemini` instances and 8 distinct `genai.Client` objects (+18.5 MB RSS per worker in our measurement). That alone is far too small to explain a kill, but it does mean 2.7.0 retains state where 2.6.2 let it be collected.
**Environment Details:**
- ADK version: 2.7.0 (working: 2.6.2)
- google-cloud-aiplatform: 1.164.0 (working: 1.163.0)
- Python: 3.13
- Runtime: Vertex AI Agent Engine (`reasoningEngines`), region `asia-northeast1`, 11 uvicorn workers
- Model: `gemini-2.5-flash` / `gemini-3.5-flash`, `stream: False`, Vertex backend
- Telemetry: `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`
- Plugins: `BigQueryAgentAnalyticsPlugin`
Happy to provide the full NDJSON stream, additional log windows, or to run a targeted experiment (e.g. with telemetry disabled) if that would help narrow it down.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.