googleapis / googleapis/python-aiplatform
Per-request `_force_flush_otel()` in ADK template blocks streaming responses
- Lenguaje dominante
- Python
- Estrellas
- 905
- Forks
- 465
- Merge medio
- 1 d 13 h
- PR fusionados (30 d)
- 44
Descripción
### Environment
- `google-cloud-aiplatform` (vertexai SDK)
- `vertexai/agent_engines/templates/adk.py`
- Agent Runtime with `identity_type=AGENT_IDENTITY`
- ADK >= 1.17.0
### Description
#### Problem
The ADK template's `streaming_agent_run_with_events` and `async_stream_query` methods call `_force_flush_otel()` in their `finally` blocks on **every request**. This triggers a synchronous `BatchSpanProcessor.force_flush()` that blocks the response stream until all queued spans are exported to `telemetry.googleapis.com`.
In Agent Identity environments where the export uses certificate-bound tokens (mTLS via SPIFFE/WIF), the authentication overhead is significantly higher than standard OAuth, and the blocking duration becomes noticeable to end users. Setting `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` eliminates the overhead, confirming the flush is the sole cause.
#### Root Cause
`_force_flush_otel()` in `adk.py` calls `tracer_provider.force_flush()` via `asyncio.to_thread()`, which invokes `BatchSpanProcessor._export(BatchExportStrategy.EXPORT_ALL)` — a blocking synchronous export. The default `OTEL_BSP_EXPORT_TIMEOUT` is 30,000ms, so the flush can block for up to 30 seconds depending on the mTLS authentication latency.
The comment on the call site reads:
```
# Avoid telemetry data loss having to do with CPU throttling on instance turndown
```
This concern is valid, but per-request flush is the wrong mechanism for it. `BatchSpanProcessor` already exports asynchronously at a configurable interval (default 5s). Telemetry data loss on shutdown should be handled by a shutdown hook, not by blocking every request.
#### Reproduction
1. Deploy an ADK agent to Agent Runtime with `identity_type=AGENT_IDENTITY`
2. Ensure telemetry is enabled (`GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`, or leave at default for ADK >= 1.17)
3. Send a streaming query
4. Observe a significant gap between agent execution completion and response delivery
5. Set `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` and repeat — the gap disappears
#### Expected Behavior
The OTel flush should not block the request path. `BatchSpanProcessor` is designed to export spans asynchronously in background threads. Per-request `force_flush()` negates this design and introduces user-facing latency proportional to the telemetry export duration.
When running the same agent on Cloud Run with ADK `Runner.run_async()` directly (bypassing the ADK template wrapper), no such overhead exists — the agent execution time is the same, but the framework-level gap is negligible.
#### Code References
**Call sites** (per-request flush in `finally` blocks):
- `streaming_agent_run_with_events`
- `async_stream_query`
**Flush implementation**:
- `_force_flush_otel`
### Workaround
Set `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=false` in the Agent Runtime deployment's `env_vars`. This disables Cloud Trace export but does not affect BigQuery Agent Analytics Plugin (which uses its own data path).
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.