google / google/adk-python

2.7.0 on Agent Engine: worker dies without a traceback on the second model call after a tool round-trip, truncating the stream

Đang mở
#6,757 5 bình luận 0 reaction 1 người được giao Được @llalitkumarrr nhận Xem trên GitHub
agent engine request clarification
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
31

Mô tả

## 🔴 Required Information

**Bug description:**

After upgrading `google-adk` 2.6.2 → 2.7.0 (with `google-cloud-aiplatform` 1.163.0 → 1.164.0), every invocation that involves a tool call fails on Vertex AI Agent Engine. The uvicorn worker process handling the request **disappears without emitting a Python traceback**, and the parent process starts a replacement worker a few seconds later. The `:streamQuery` response ends after two events, so the client sees a well-formed HTTP 200 stream that simply contains no model text.

The invocation dies **after** the tool round-trip completes successfully — the failure is in the second model call of the turn.

Reverting only the two dependencies (no application code change) makes the same question complete normally.

**Steps to Reproduce:**

1. Build an `LlmAgent` as the root agent (`coordinator`) with:
- a function tool that performs network I/O (in our case a Workspace search returning ~5 results),
- seven `sub_agents`, all `mode="single_turn"` (five domain specialists, one artifact reader, one grounding judge),
- `before_model_callback` that sets `tool_config` (`FunctionCallingConfigMode.ANY`) on the first LLM call of a turn,
- `after_agent_callback` returning `types.Content` when the root produced no visible text,
- `BigQueryAgentAnalyticsPlugin` registered via `AdkApp(agent=..., plugins=[...])`.
2. Deploy to Agent Engine (`vertexai.agent_engines`, `agent_framework="google-adk"`), region `asia-northeast1`, Python 3.13.
3. Send a question through `:streamQuery` that requires the tool.

**Expected Behavior:**

The turn proceeds: model → tool call → tool result → model → delegation to a sub-agent → final text. This is exactly what 2.6.2 does (14 events, final answer with citations).

**Observed Behavior:**

The stream ends after 2 events:

```jsonc
// event 1 — coordinator, function_call + thought_signature
{"author":"coordinator","content":{"parts":[{"function_call":{"name":"search_workspace_documents", ...},
"thought_signature":"..."}],"role":"model"},"finish_reason":"STOP", ...}

// event 2 — coordinator, function_response (tool succeeded, 5 results)
{"author":"coordinator","content":{"parts":[{"function_response":{"name":"search_workspace_documents",
"response":{"status":"success","results":[ ... 5 items ... ]}}}],"role":"user"}, ...}

// nothing further
```

Runtime logs for the same invocation:

```
03:40:04.415 [26] POST /api/stream_reasoning_engine HTTP/1.1" 200 OK
03:40:06.514 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:07.692 [26] Response received from the model.
(tool executes successfully, ~5.4 s)
03:40:13.100 [26] Sending out request, model: gemini-2.5-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
03:40:17.518 [413] gcsfs experimental features enabled ... <- different PID: a new worker booted
03:40:17.529 [413] Plugin 'bigquery_agent_analytics' registered.
03:40:17.532 [413] Application startup complete.
```

Worker `[26]` never logs `Response received from the model.` for the second call. **No traceback, no ERROR-level entry** appears anywhere in the invocation window, which points to the process being killed by a signal rather than raising.

**What we ruled out:**

| Hypothesis | Result |
| --- | --- |
| Model generation | Reproduced with both `gemini-2.5-flash` and `gemini-3.5-flash` |
| The tool itself | Tool completed in 5.4 s and returned `status: success` with 5 results |
| Auth / credentials | `status: success`, no `auth_required` |
| Application code | Unchanged across the working and failing deploys; only the two dependencies differ |
| `tool_config` forcing not being released | Verified with a real `Runner` on 2.7.0: the second call has `tool_config=None` as expected |
| Dependency set growth | `uv export --only-group runtime` diff is 3 lines (adk, aiplatform, and `google-cloud-iam` dropped); no new packages |

**Possibly related observation (not a proven cause):**

Container memory allocation for the same question roughly doubled: ~0.8 GiB on 2.6.2 vs ~1.6–2.0 GiB on 2.7.0 (Agent Engine console metric).

We also measured that `LlmAgent._resolved_model` (new in 2.7.0) keeps one resolved `BaseLlm` per agent. With 8 agents that all pass the model as a **string**, this retains 8 distinct `Gemini` instances and 8 distinct `genai.Client` objects (+18.5 MB RSS per worker in our measurement). That alone is far too small to explain a kill, but it does mean 2.7.0 retains state where 2.6.2 let it be collected.

**Environment Details:**

- ADK version: 2.7.0 (working: 2.6.2)
- google-cloud-aiplatform: 1.164.0 (working: 1.163.0)
- Python: 3.13
- Runtime: Vertex AI Agent Engine (`reasoningEngines`), region `asia-northeast1`, 11 uvicorn workers
- Model: `gemini-2.5-flash` / `gemini-3.5-flash`, `stream: False`, Vertex backend
- Telemetry: `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`
- Plugins: `BigQueryAgentAnalyticsPlugin`

Happy to provide the full NDJSON stream, additional log windows, or to run a targeted experiment (e.g. with telemetry disabled) if that would help narrow it down.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.