google / google/adk-python

ApiServer.get_fast_api_app() registers unbounded in-memory span exporters (memory leak on long-lived servers)

Abierto
#6,692 4 comentarios 0 reacciones 1 asignado Reclamado por @sanketpatil06 Ver en GitHub
tracing
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

## 🔴 Required Information

**Bug description:**

`ApiServer.get_fast_api_app()` unconditionally registers two in-memory span pipelines on the tracer provider:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L1048-L1057

`InMemoryExporter.export()` appends to an unbounded list, and `ApiServerSpanExporter` writes every span's attributes into `trace_dict`, which is never pruned:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L418-L432

`InMemoryExporter.clear()` is the only way to release them and has no callers anywhere in the package:

https://github.com/google/adk-python/blob/a56f6e13ae38296b608808c7a3b37efe4b8c862e/src/google/adk/cli/api_server.py#L444-L445

Every span the process creates is therefore retained until the process dies.

The only reader of this data seems's the dev UI trace endpoint, which lives on `DevServer` (`dev_server.py:567` calls `memory_exporter.get_finished_spans`).

`_setup_telemetry` installs these processors on the global tracer provider, so the retention covers every span in the process, not only ADK's own.

There is no way to opt out: `otel_to_cloud` only adds GCP exporters on top, and `internal_exporters` is not exposed through `get_fast_api_app()`.

**Steps to Reproduce:**

1. `pip install google-adk` (reproduced on 2.5.0).
2. Serve any agent through `ApiServer.get_fast_api_app()`.
3. Generate spans. Agent invocations are enough on their own; installing `opentelemetry-instrumentation-fastapi` and calling `FastAPIInstrumentor.instrument_app(app)` makes it far quicker to observe, since every HTTP request then adds ~3 spans.
4. Watch process RSS and `len(server._memory_exporter._spans)` grow linearly, without bound.

**Expected Behavior:**

Spans are released after export, and server memory reaches a steady state under a constant request rate. Debug/trace retention is bounded, cleared, or only registered where something reads it.

**Observed Behavior:**

Linear, unbounded memory growth for the life of the process. Measured on `python:3.13-slim` (linux/amd64), serving only trivial health-probe requests at ~4 requests/s for 20 minutes, with `opentelemetry-instrumentation-fastapi` supplying ~3 spans per request.

The instrumentation only sets the rate. Agent invocation spans accumulate the same way with no HTTP instrumentation installed.

**Environment Details:**

- ADK Library Version: 2.5.0
- Desktop OS: Linux (`python:3.13-slim` container, linux/amd64)
- Python Version: 3.13
- `opentelemetry-instrumentation-fastapi`: 0.63b1

**Model Information:**

- Are you using LiteLLM: No
- Which model is being used: N/A, not model-related; the leak reproduces with zero LLM calls.

---

## 🟡 Optional Information

**Additional Context:**

- #2792 asks for a way to disable or extend the internal OTel tracing for a different reason (context propagation conflicts). A supported opt-out would resolve both.
- Possible fixes: register these exporters only in `DevServer`, expose an opt-out parameter on `get_fast_api_app()`, or bound the retention (`collections.deque(maxlen=...)` and a capped `trace_dict`).

**Minimal Reproduction Code:**

```python
import fastapi.testclient
import opentelemetry.instrumentation.fastapi
import opentelemetry.trace
from google.adk.cli.api_server import InMemoryExporter
from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(agents_dir="path/to/empty/dir", web=False)
opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app(app)

processors = (
opentelemetry.trace.get_tracer_provider()._active_span_processor._span_processors
)
memory_exporter = next(
p.span_exporter
for p in processors
if isinstance(getattr(p, "span_exporter", None), InMemoryExporter)
)

client = fastapi.testclient.TestClient(app)
print("spans before:", len(memory_exporter._spans))
for _ in range(1_000):
client.get("/health")
print("spans after 1000 requests:", len(memory_exporter._spans))
```

**How often has this issue occurred?:**

- Always (100%)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.