googleapis / googleapis/python-aiplatform

Per-request `_force_flush_otel()` in ADK template blocks streaming responses

未关闭
#6,807 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

### Environment

- `google-cloud-aiplatform` (vertexai SDK)
- `vertexai/agent_engines/templates/adk.py`
- Agent Runtime with `identity_type=AGENT_IDENTITY`
- ADK >= 1.17.0

### Description

#### Problem

The ADK template's `streaming_agent_run_with_events` and `async_stream_query` methods call `_force_flush_otel()` in their `finally` blocks on **every request**. This triggers a synchronous `BatchSpanProcessor.force_flush()` that blocks the response stream until all queued spans are exported to `telemetry.googleapis.com`.

In Agent Identity environments where the export uses certificate-bound tokens (mTLS via SPIFFE/WIF), the authentication overhead is significantly higher than standard OAuth, and the blocking duration becomes noticeable to end users. Setting `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` eliminates the overhead, confirming the flush is the sole cause.

#### Root Cause

`_force_flush_otel()` in `adk.py` calls `tracer_provider.force_flush()` via `asyncio.to_thread()`, which invokes `BatchSpanProcessor._export(BatchExportStrategy.EXPORT_ALL)` — a blocking synchronous export. The default `OTEL_BSP_EXPORT_TIMEOUT` is 30,000ms, so the flush can block for up to 30 seconds depending on the mTLS authentication latency.

The comment on the call site reads:
```
# Avoid telemetry data loss having to do with CPU throttling on instance turndown
```

This concern is valid, but per-request flush is the wrong mechanism for it. `BatchSpanProcessor` already exports asynchronously at a configurable interval (default 5s). Telemetry data loss on shutdown should be handled by a shutdown hook, not by blocking every request.

#### Reproduction

1. Deploy an ADK agent to Agent Runtime with `identity_type=AGENT_IDENTITY`
2. Ensure telemetry is enabled (`GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`, or leave at default for ADK >= 1.17)
3. Send a streaming query
4. Observe a significant gap between agent execution completion and response delivery
5. Set `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` and repeat — the gap disappears

#### Expected Behavior

The OTel flush should not block the request path. `BatchSpanProcessor` is designed to export spans asynchronously in background threads. Per-request `force_flush()` negates this design and introduces user-facing latency proportional to the telemetry export duration.

When running the same agent on Cloud Run with ADK `Runner.run_async()` directly (bypassing the ADK template wrapper), no such overhead exists — the agent execution time is the same, but the framework-level gap is negligible.

#### Code References

**Call sites** (per-request flush in `finally` blocks):
- `streaming_agent_run_with_events`
- `async_stream_query`

**Flush implementation**:
- `_force_flush_otel`

### Workaround

Set `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` in the Agent Runtime deployment's `env_vars`. This disables Cloud Trace export but does not affect BigQuery Agent Analytics Plugin (which uses its own data path).

贡献指南

打开贡献指南

调研方向

从 vertexai/agent_engines/templates/adk.py 开始,检查 streaming_agent_run_with_events、async_stream_query 和 _force_flush_otel,然后分别在启用和禁用遥测的情况下重现一次流式查询。完成的标准是:响应交付不再等待每个请求的遥测刷新,同时遥测继续异步导出,并且关机时的数据丢失处理仍得到解决。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend, observability-sre
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
描述清楚
新手友好度
68/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。