google / google/adk-python

ApiServer.get_fast_api_app() registers unbounded in-memory span exporters (memory leak on long-lived servers)

オープン
#6,692 コメント 4 件 リアクション 0 件 担当者 1 名 @sanketpatil06 が担当を希望しています GitHub で見る
tracing
主要言語
Python
スター
21.5k
フォーク
4k
平均マージ
1日 14時間
マージ済み PR(30日)
37

説明

## 🔴 Required Information

**Bug description:**

`ApiServer.get_fast_api_app()` unconditionally registers two in-memory span pipelines on the tracer provider:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L1048-L1057

`InMemoryExporter.export()` appends to an unbounded list, and `ApiServerSpanExporter` writes every span's attributes into `trace_dict`, which is never pruned:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L418-L432

`InMemoryExporter.clear()` is the only way to release them and has no callers anywhere in the package:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L444-L445

Every span the process creates is therefore retained until the process dies.

The only reader of this data seems's the dev UI trace endpoint, which lives on `DevServer` (`dev_server.py:567` calls `memory_exporter.get_finished_spans`).

`_setup_telemetry` installs these processors on the global tracer provider, so the retention covers every span in the process, not only ADK's own.

There is no way to opt out: `otel_to_cloud` only adds GCP exporters on top, and `internal_exporters` is not exposed through `get_fast_api_app()`.

**Steps to Reproduce:**

1. `pip install google-adk` (reproduced on 2.5.0).
2. Serve any agent through `ApiServer.get_fast_api_app()`.
3. Generate spans. Agent invocations are enough on their own; installing `opentelemetry-instrumentation-fastapi` and calling `FastAPIInstrumentor.instrument_app(app)` makes it far quicker to observe, since every HTTP request then adds ~3 spans.
4. Watch process RSS and `len(server._memory_exporter._spans)` grow linearly, without bound.

**Expected Behavior:**

Spans are released after export, and server memory reaches a steady state under a constant request rate. Debug/trace retention is bounded, cleared, or only registered where something reads it.

**Observed Behavior:**

Linear, unbounded memory growth for the life of the process. Measured on `python:3.13-slim` (linux/amd64), serving only trivial health-probe requests at ~4 requests/s for 20 minutes, with `opentelemetry-instrumentation-fastapi` supplying ~3 spans per request.

The instrumentation only sets the rate. Agent invocation spans accumulate the same way with no HTTP instrumentation installed.

**Environment Details:**

- ADK Library Version: 2.5.0
- Desktop OS: Linux (`python:3.13-slim` container, linux/amd64)
- Python Version: 3.13
- `opentelemetry-instrumentation-fastapi`: 0.63b1

**Model Information:**

- Are you using LiteLLM: No
- Which model is being used: N/A, not model-related; the leak reproduces with zero LLM calls.

---

## 🟡 Optional Information

**Additional Context:**

- #2792 asks for a way to disable or extend the internal OTel tracing for a different reason (context propagation conflicts). A supported opt-out would resolve both.
- Possible fixes: register these exporters only in `DevServer`, expose an opt-out parameter on `get_fast_api_app()`, or bound the retention (`collections.deque(maxlen=...)` and a capped `trace_dict`).

**Minimal Reproduction Code:**

```python
import fastapi.testclient
import opentelemetry.instrumentation.fastapi
import opentelemetry.trace
from google.adk.cli.api_server import InMemoryExporter
from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(agents_dir="path/to/empty/dir", web=False)
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app(app)

processors = (
opentelemetry.trace.get_tracer_provider()._active_span_processor._span_processors
)
memory_exporter = next(
p.span_exporter
for p in processors
if isinstance(getattr(p, "span_exporter", None), InMemoryExporter)
)

client = fastapi.testclient.TestClient(app)
print("spans before:", len(memory_exporter._spans))
for _ in range(1_000):
client.get("/health")
print("spans after 1000 requests:", len(memory_exporter._spans))
```

**How often has this issue occurred?:**

- Always (100%)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。