google / google/adk-python

Cloud telemetry export fails with invalid_scope when ADC is a service-account key file

Abierto
#7,024 2 comentarios 0 reacciones 1 asignado Reclamado por @sanketpatil06 Ver en GitHub
request clarification tracing
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

### Summary

With `get_fast_api_app(..., otel_to_cloud=True)` and Application Default Credentials supplied as a **service-account key file**, every Cloud Trace and Cloud Monitoring export fails at the token grant, before any HTTP request is made:

```
google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', ...)
... opentelemetry/exporter/otlp/proto/http/{trace,metric}_exporter/__init__.py
... google/auth/transport/requests.py request
... google/oauth2/service_account.py _perform_refresh_token
... google/oauth2/_client.py jwt_grant
```

The application itself is unaffected — only the exporters fail — so the symptom is a continuous stream of tracebacks in the container log (observed at roughly nine per minute) with no telemetry arriving and no other outward sign.

### Environment

* `google-adk` 2.8.0, `google-auth` 2.57.0, Python 3.12
* Host **not** on GCP (self-hosted container), ADC via `GOOGLE_APPLICATION_CREDENTIALS=/path/key.json`
* `GOOGLE_CLOUD_PROJECT` set

### Cause

ADC is resolved without scopes and the result is handed to the OTLP exporters' `AuthorizedSession`. A service-account key credential has `requires_scopes = True` and no scopes, so the JWT grant is rejected.

Two call sites resolve it that way:

* `google/adk/cli/api_server.py:700` — `_setup_gcp_telemetry`: `credentials, project_id = google.auth.default()`, then passed as `google_auth=` to `get_gcp_exporters`. **This is the one that fires on the `get_fast_api_app` path**, because it supplies `google_auth=` and the fallback below is never reached.
* `google/adk/telemetry/google_cloud.py:96` and `:201` — the `google_auth is None` fallback inside `get_gcp_exporters` / the metrics helper.

It does not reproduce on GCE, Cloud Run, GKE or Agent Engine: the metadata server issues a usable token regardless of the scopes request, so the omission is invisible there. A key file is what exposes it.

### Minimal reproduction

```python
import google.auth
from google.auth.transport.requests import Request

# GOOGLE_APPLICATION_CREDENTIALS -> a service-account key file
creds, project = google.auth.default()
creds.requires_scopes # True
creds.refresh(Request()) # RefreshError: invalid_scope

creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
creds.refresh(Request()) # OK
```

### Suggested fix

Apply scopes where they are needed, leaving credentials that do not need them untouched:

```python
from google.auth.credentials import with_scopes_if_required

credentials = with_scopes_if_required(
credentials, ["https://www.googleapis.com/auth/cloud-platform"]
)
```

or resolve with `google.auth.default(scopes=[...])`, which routes through the same helper.

Worth noting for whichever is chosen: `with_scopes_if_required` rewrites any credential whose `requires_scopes` is true, and that includes `google.auth.compute_engine.Credentials` — measured `requires_scopes is True` on 2.57.0. On GCE the metadata server ignores the scopes parameter, but on Cloud Run / App Engine Flex it honours it, so this is not a no-op everywhere in-cloud even though it is harmless.

### Workaround

Build the exporters in the application instead of letting `otel_to_cloud=True` resolve ADC, and pass `otel_to_cloud=False` so a second, unscoped set is not installed:

```python
creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
hooks = get_gcp_exporters(
enable_cloud_tracing=True, enable_cloud_metrics=True, enable_cloud_logging=True,
google_auth=(creds, project),
)
maybe_set_otel_providers([hooks], otel_resource=get_gcp_resource(project))
```

Two things have to be reproduced by hand when doing this, and both are easy to miss:

* `_setup_instrumentation_lib_if_installed` — without `GoogleGenAiSdkInstrumentor` the export carries agent spans and no model calls (plus the HTTPX/gRPC instrumentors on Agent Engine).
* `maybe_install_request_metrics_middleware`, which `get_fast_api_app` skips under `otel_to_cloud=False`. On Agent Engine that is not just a flush-timing loss: `_RequestDrivenMetricReader` has no background ticker, `note_request_start` is the only writer of its in-flight counter, and the generate_content path refuses to collect while that counter is zero — so metrics stop entirely.

Possibly related: discussion #1009 reports the same error with a key file.

### Related, same off-GCP class (observed after applying the workaround above)

Once the token grant succeeds, metric export from a non-GCP host is still rejected by `telemetry.googleapis.com/v1/metrics` with `400 INVALID_ARGUMENT: prometheus_target resource type must have an instance specified`. Off Agent Engine, `get_gcp_resource()` builds a bare `Resource({"gcp.project_id": ...})`; the Telemetry API's Prometheus mapping requires `instance` (from `service.instance.id`) and `location` on the resource, and the GCP resource detector supplies neither outside Google Cloud. Setting `OTEL_RESOURCE_ATTRIBUTES=service.instance.id=,location=` makes the same export return 200, so it is only the resource that is missing. Traces are unaffected. Mentioned here because it hits the same deployment shape (key file, self-hosted) right after the scope problem is fixed; happy to open it separately if preferred.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.