google / google/adk-python

ApiServer.get_fast_api_app() registers unbounded in-memory span exporters (memory leak on long-lived servers)

Đang mở
#6,692 4 bình luận 0 reaction 1 người được giao Được @sanketpatil06 nhận Xem trên GitHub
tracing
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
31

Mô tả

## 🔴 Required Information

**Bug description:**

`ApiServer.get_fast_api_app()` unconditionally registers two in-memory span pipelines on the tracer provider:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L1048-L1057

`InMemoryExporter.export()` appends to an unbounded list, and `ApiServerSpanExporter` writes every span's attributes into `trace_dict`, which is never pruned:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L418-L432

`InMemoryExporter.clear()` is the only way to release them and has no callers anywhere in the package:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L444-L445

Every span the process creates is therefore retained until the process dies.

The only reader of this data seems's the dev UI trace endpoint, which lives on `DevServer` (`dev_server.py:567` calls `memory_exporter.get_finished_spans`).

`_setup_telemetry` installs these processors on the global tracer provider, so the retention covers every span in the process, not only ADK's own.

There is no way to opt out: `otel_to_cloud` only adds GCP exporters on top, and `internal_exporters` is not exposed through `get_fast_api_app()`.

**Steps to Reproduce:**

1. `pip install google-adk` (reproduced on 2.5.0).
2. Serve any agent through `ApiServer.get_fast_api_app()`.
3. Generate spans. Agent invocations are enough on their own; installing `opentelemetry-instrumentation-fastapi` and calling `FastAPIInstrumentor.instrument_app(app)` makes it far quicker to observe, since every HTTP request then adds ~3 spans.
4. Watch process RSS and `len(server._memory_exporter._spans)` grow linearly, without bound.

**Expected Behavior:**

Spans are released after export, and server memory reaches a steady state under a constant request rate. Debug/trace retention is bounded, cleared, or only registered where something reads it.

**Observed Behavior:**

Linear, unbounded memory growth for the life of the process. Measured on `python:3.13-slim` (linux/amd64), serving only trivial health-probe requests at ~4 requests/s for 20 minutes, with `opentelemetry-instrumentation-fastapi` supplying ~3 spans per request.

The instrumentation only sets the rate. Agent invocation spans accumulate the same way with no HTTP instrumentation installed.

**Environment Details:**

- ADK Library Version: 2.5.0
- Desktop OS: Linux (`python:3.13-slim` container, linux/amd64)
- Python Version: 3.13
- `opentelemetry-instrumentation-fastapi`: 0.63b1

**Model Information:**

- Are you using LiteLLM: No
- Which model is being used: N/A, not model-related; the leak reproduces with zero LLM calls.

---

## 🟡 Optional Information

**Additional Context:**

- #2792 asks for a way to disable or extend the internal OTel tracing for a different reason (context propagation conflicts). A supported opt-out would resolve both.
- Possible fixes: register these exporters only in `DevServer`, expose an opt-out parameter on `get_fast_api_app()`, or bound the retention (`collections.deque(maxlen=...)` and a capped `trace_dict`).

**Minimal Reproduction Code:**

```python
import fastapi.testclient
import opentelemetry.instrumentation.fastapi
import opentelemetry.trace
from google.adk.cli.api_server import InMemoryExporter
from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(agents_dir="path/to/empty/dir", web=False)
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app(app)

processors = (
opentelemetry.trace.get_tracer_provider()._active_span_processor._span_processors
)
memory_exporter = next(
p.span_exporter
for p in processors
if isinstance(getattr(p, "span_exporter", None), InMemoryExporter)
)

client = fastapi.testclient.TestClient(app)
print("spans before:", len(memory_exporter._spans))
for _ in range(1_000):
client.get("/health")
print("spans after 1000 requests:", len(memory_exporter._spans))
```

**How often has this issue occurred?:**

- Always (100%)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.