google / google/adk-python

ApiServer.get_fast_api_app() registers unbounded in-memory span exporters (memory leak on long-lived servers)

Ouverte
#6,692 4 commentaires 0 réactions 1 personne assignée Réclamée par @sanketpatil06 Voir sur GitHub
tracing
Langage dominant
Python
Étoiles
21.5k
Forks
4k
Merge moyen
1 j 22 h
PR mergées (30 j)
31

Description

## 🔴 Required Information

**Bug description:**

`ApiServer.get_fast_api_app()` unconditionally registers two in-memory span pipelines on the tracer provider:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L1048-L1057

`InMemoryExporter.export()` appends to an unbounded list, and `ApiServerSpanExporter` writes every span's attributes into `trace_dict`, which is never pruned:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L418-L432

`InMemoryExporter.clear()` is the only way to release them and has no callers anywhere in the package:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L444-L445

Every span the process creates is therefore retained until the process dies.

The only reader of this data seems's the dev UI trace endpoint, which lives on `DevServer` (`dev_server.py:567` calls `memory_exporter.get_finished_spans`).

`_setup_telemetry` installs these processors on the global tracer provider, so the retention covers every span in the process, not only ADK's own.

There is no way to opt out: `otel_to_cloud` only adds GCP exporters on top, and `internal_exporters` is not exposed through `get_fast_api_app()`.

**Steps to Reproduce:**

1. `pip install google-adk` (reproduced on 2.5.0).
2. Serve any agent through `ApiServer.get_fast_api_app()`.
3. Generate spans. Agent invocations are enough on their own; installing `opentelemetry-instrumentation-fastapi` and calling `FastAPIInstrumentor.instrument_app(app)` makes it far quicker to observe, since every HTTP request then adds ~3 spans.
4. Watch process RSS and `len(server._memory_exporter._spans)` grow linearly, without bound.

**Expected Behavior:**

Spans are released after export, and server memory reaches a steady state under a constant request rate. Debug/trace retention is bounded, cleared, or only registered where something reads it.

**Observed Behavior:**

Linear, unbounded memory growth for the life of the process. Measured on `python:3.13-slim` (linux/amd64), serving only trivial health-probe requests at ~4 requests/s for 20 minutes, with `opentelemetry-instrumentation-fastapi` supplying ~3 spans per request.

The instrumentation only sets the rate. Agent invocation spans accumulate the same way with no HTTP instrumentation installed.

**Environment Details:**

- ADK Library Version: 2.5.0
- Desktop OS: Linux (`python:3.13-slim` container, linux/amd64)
- Python Version: 3.13
- `opentelemetry-instrumentation-fastapi`: 0.63b1

**Model Information:**

- Are you using LiteLLM: No
- Which model is being used: N/A, not model-related; the leak reproduces with zero LLM calls.

---

## 🟡 Optional Information

**Additional Context:**

- #2792 asks for a way to disable or extend the internal OTel tracing for a different reason (context propagation conflicts). A supported opt-out would resolve both.
- Possible fixes: register these exporters only in `DevServer`, expose an opt-out parameter on `get_fast_api_app()`, or bound the retention (`collections.deque(maxlen=...)` and a capped `trace_dict`).

**Minimal Reproduction Code:**

```python
import fastapi.testclient
import opentelemetry.instrumentation.fastapi
import opentelemetry.trace
from google.adk.cli.api_server import InMemoryExporter
from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(agents_dir="path/to/empty/dir", web=False)
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app(app)

processors = (
opentelemetry.trace.get_tracer_provider()._active_span_processor._span_processors
)
memory_exporter = next(
p.span_exporter
for p in processors
if isinstance(getattr(p, "span_exporter", None), InMemoryExporter)
)

client = fastapi.testclient.TestClient(app)
print("spans before:", len(memory_exporter._spans))
for _ in range(1_000):
client.get("/health")
print("spans after 1000 requests:", len(memory_exporter._spans))
```

**How often has this issue occurred?:**

- Always (100%)

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.