ApiServer.get_fast_api_app() registers unbounded in-memory span exporters (memory leak on long-lived servers)
- Ngôn ngữ chính
- Python
- Star
- 21.5k
- Fork
- 4k
- Merge trung bình
- 1 ngày 22 giờ
- Pull request đã merge (30 ngày)
- 31
Mô tả
## 🔴 Required Information
**Bug description:**
`ApiServer.get_fast_api_app()` unconditionally registers two in-memory span pipelines on the tracer provider:
https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L1048-L1057
`InMemoryExporter.export()` appends to an unbounded list, and `ApiServerSpanExporter` writes every span's attributes into `trace_dict`, which is never pruned:
https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L418-L432
`InMemoryExporter.clear()` is the only way to release them and has no callers anywhere in the package:
https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L444-L445
Every span the process creates is therefore retained until the process dies.
The only reader of this data seems's the dev UI trace endpoint, which lives on `DevServer` (`dev_server.py:567` calls `memory_exporter.get_finished_spans`).
`_setup_telemetry` installs these processors on the global tracer provider, so the retention covers every span in the process, not only ADK's own.
There is no way to opt out: `otel_to_cloud` only adds GCP exporters on top, and `internal_exporters` is not exposed through `get_fast_api_app()`.
**Steps to Reproduce:**
1. `pip install google-adk` (reproduced on 2.5.0).
2. Serve any agent through `ApiServer.get_fast_api_app()`.
3. Generate spans. Agent invocations are enough on their own; installing `opentelemetry-instrumentation-fastapi` and calling `FastAPIInstrumentor.instrument_app(app)` makes it far quicker to observe, since every HTTP request then adds ~3 spans.
4. Watch process RSS and `len(server._memory_exporter._spans)` grow linearly, without bound.
**Expected Behavior:**
Spans are released after export, and server memory reaches a steady state under a constant request rate. Debug/trace retention is bounded, cleared, or only registered where something reads it.
**Observed Behavior:**
Linear, unbounded memory growth for the life of the process. Measured on `python:3.13-slim` (linux/amd64), serving only trivial health-probe requests at ~4 requests/s for 20 minutes, with `opentelemetry-instrumentation-fastapi` supplying ~3 spans per request.
The instrumentation only sets the rate. Agent invocation spans accumulate the same way with no HTTP instrumentation installed.
**Environment Details:**
- ADK Library Version: 2.5.0
- Desktop OS: Linux (`python:3.13-slim` container, linux/amd64)
- Python Version: 3.13
- `opentelemetry-instrumentation-fastapi`: 0.63b1
**Model Information:**
- Are you using LiteLLM: No
- Which model is being used: N/A, not model-related; the leak reproduces with zero LLM calls.
---
## 🟡 Optional Information
**Additional Context:**
- #2792 asks for a way to disable or extend the internal OTel tracing for a different reason (context propagation conflicts). A supported opt-out would resolve both.
- Possible fixes: register these exporters only in `DevServer`, expose an opt-out parameter on `get_fast_api_app()`, or bound the retention (`collections.deque(maxlen=...)` and a capped `trace_dict`).
**Minimal Reproduction Code:**
```python
import fastapi.testclient
import opentelemetry.instrumentation.fastapi
import opentelemetry.trace
from google.adk.cli.api_server import InMemoryExporter
from google.adk.cli.fast_api import get_fast_api_app
app = get_fast_api_app(agents_dir="path/to/empty/dir", web=False)
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app(app)
processors = (
opentelemetry.trace.get_tracer_provider()._active_span_processor._span_processors
)
memory_exporter = next(
p.span_exporter
for p in processors
if isinstance(getattr(p, "span_exporter", None), InMemoryExporter)
)
client = fastapi.testclient.TestClient(app)
print("spans before:", len(memory_exporter._spans))
for _ in range(1_000):
client.get("/health")
print("spans after 1000 requests:", len(memory_exporter._spans))
```
**How often has this issue occurred?:**
- Always (100%)
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.