google / google/adk-python

Cloud telemetry export fails with invalid_scope when ADC is a service-account key file

オープン
#7,024 コメント 2 件 リアクション 0 件 担当者 1 名 @sanketpatil06 が担当を希望しています GitHub で見る
request clarification tracing
主要言語
Python
スター
21.5k
フォーク
4k
平均マージ
1日 14時間
マージ済み PR(30日)
37

説明

### Summary

With `get_fast_api_app(..., otel_to_cloud=True)` and Application Default Credentials supplied as a **service-account key file**, every Cloud Trace and Cloud Monitoring export fails at the token grant, before any HTTP request is made:

```
google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', ...)
... opentelemetry/exporter/otlp/proto/http/{trace,metric}_exporter/__init__.py
... google/auth/transport/requests.py request
... google/oauth2/service_account.py _perform_refresh_token
... google/oauth2/_client.py jwt_grant
```

The application itself is unaffected — only the exporters fail — so the symptom is a continuous stream of tracebacks in the container log (observed at roughly nine per minute) with no telemetry arriving and no other outward sign.

### Environment

* `google-adk` 2.8.0, `google-auth` 2.57.0, Python 3.12
* Host **not** on GCP (self-hosted container), ADC via `GOOGLE_APPLICATION_CREDENTIALS=/path/key.json`
* `GOOGLE_CLOUD_PROJECT` set

### Cause

ADC is resolved without scopes and the result is handed to the OTLP exporters' `AuthorizedSession`. A service-account key credential has `requires_scopes = True` and no scopes, so the JWT grant is rejected.

Two call sites resolve it that way:

* `google/adk/cli/api_server.py:700` — `_setup_gcp_telemetry`: `credentials, project_id = google.auth.default()`, then passed as `google_auth=` to `get_gcp_exporters`. **This is the one that fires on the `get_fast_api_app` path**, because it supplies `google_auth=` and the fallback below is never reached.
* `google/adk/telemetry/google_cloud.py:96` and `:201` — the `google_auth is None` fallback inside `get_gcp_exporters` / the metrics helper.

It does not reproduce on GCE, Cloud Run, GKE or Agent Engine: the metadata server issues a usable token regardless of the scopes request, so the omission is invisible there. A key file is what exposes it.

### Minimal reproduction

```python
import google.auth
from google.auth.transport.requests import Request

# GOOGLE_APPLICATION_CREDENTIALS -> a service-account key file
creds, project = google.auth.default()
creds.requires_scopes # True
creds.refresh(Request()) # RefreshError: invalid_scope

creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
creds.refresh(Request()) # OK
```

### Suggested fix

Apply scopes where they are needed, leaving credentials that do not need them untouched:

```python
from google.auth.credentials import with_scopes_if_required

credentials = with_scopes_if_required(
credentials, ["https://www.googleapis.com/auth/cloud-platform"]
)
```

or resolve with `google.auth.default(scopes=[...])`, which routes through the same helper.

Worth noting for whichever is chosen: `with_scopes_if_required` rewrites any credential whose `requires_scopes` is true, and that includes `google.auth.compute_engine.Credentials` — measured `requires_scopes is True` on 2.57.0. On GCE the metadata server ignores the scopes parameter, but on Cloud Run / App Engine Flex it honours it, so this is not a no-op everywhere in-cloud even though it is harmless.

### Workaround

Build the exporters in the application instead of letting `otel_to_cloud=True` resolve ADC, and pass `otel_to_cloud=False` so a second, unscoped set is not installed:

```python
creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
hooks = get_gcp_exporters(
enable_cloud_tracing=True, enable_cloud_metrics=True, enable_cloud_logging=True,
google_auth=(creds, project),
)
maybe_set_otel_providers([hooks], otel_resource=get_gcp_resource(project))
```

Two things have to be reproduced by hand when doing this, and both are easy to miss:

* `_setup_instrumentation_lib_if_installed` — without `GoogleGenAiSdkInstrumentor` the export carries agent spans and no model calls (plus the HTTPX/gRPC instrumentors on Agent Engine).
* `maybe_install_request_metrics_middleware`, which `get_fast_api_app` skips under `otel_to_cloud=False`. On Agent Engine that is not just a flush-timing loss: `_RequestDrivenMetricReader` has no background ticker, `note_request_start` is the only writer of its in-flight counter, and the generate_content path refuses to collect while that counter is zero — so metrics stop entirely.

Possibly related: discussion #1009 reports the same error with a key file.

### Related, same off-GCP class (observed after applying the workaround above)

Once the token grant succeeds, metric export from a non-GCP host is still rejected by `telemetry.googleapis.com/v1/metrics` with `400 INVALID_ARGUMENT: prometheus_target resource type must have an instance specified`. Off Agent Engine, `get_gcp_resource()` builds a bare `Resource({"gcp.project_id": ...})`; the Telemetry API's Prometheus mapping requires `instance` (from `service.instance.id`) and `location` on the resource, and the GCP resource detector supplies neither outside Google Cloud. Setting `OTEL_RESOURCE_ATTRIBUTES=service.instance.id=,location=` makes the same export return 200, so it is only the resource that is missing. Traces are unaffected. Mentioned here because it hits the same deployment shape (key file, self-hosted) right after the scope problem is fixed; happy to open it separately if preferred.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。