Cloud telemetry export fails with invalid_scope when ADC is a service-account key file
- 主要语言
- Python
- 星标
- 21.5k
- 派生
- 4k
- 平均合并
- 1 天 22 小时
- 30 天内合并 PR
- 31
描述
### Summary
With `get_fast_api_app(..., otel_to_cloud=True)` and Application Default Credentials supplied as a **service-account key file**, every Cloud Trace and Cloud Monitoring export fails at the token grant, before any HTTP request is made:
```
google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', ...)
... opentelemetry/exporter/otlp/proto/http/{trace,metric}_exporter/__init__.py
... google/auth/transport/requests.py request
... google/oauth2/service_account.py _perform_refresh_token
... google/oauth2/_client.py jwt_grant
```
The application itself is unaffected — only the exporters fail — so the symptom is a continuous stream of tracebacks in the container log (observed at roughly nine per minute) with no telemetry arriving and no other outward sign.
### Environment
* `google-adk` 2.8.0, `google-auth` 2.57.0, Python 3.12
* Host **not** on GCP (self-hosted container), ADC via `GOOGLE_APPLICATION_CREDENTIALS=/path/key.json`
* `GOOGLE_CLOUD_PROJECT` set
### Cause
ADC is resolved without scopes and the result is handed to the OTLP exporters' `AuthorizedSession`. A service-account key credential has `requires_scopes = True` and no scopes, so the JWT grant is rejected.
Two call sites resolve it that way:
* `google/adk/cli/api_server.py:700` — `_setup_gcp_telemetry`: `credentials, project_id = google.auth.default()`, then passed as `google_auth=` to `get_gcp_exporters`. **This is the one that fires on the `get_fast_api_app` path**, because it supplies `google_auth=` and the fallback below is never reached.
* `google/adk/telemetry/google_cloud.py:96` and `:201` — the `google_auth is None` fallback inside `get_gcp_exporters` / the metrics helper.
It does not reproduce on GCE, Cloud Run, GKE or Agent Engine: the metadata server issues a usable token regardless of the scopes request, so the omission is invisible there. A key file is what exposes it.
### Minimal reproduction
```python
import google.auth
from google.auth.transport.requests import Request
# GOOGLE_APPLICATION_CREDENTIALS -> a service-account key file
creds, project = google.auth.default()
creds.requires_scopes # True
creds.refresh(Request()) # RefreshError: invalid_scope
creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
creds.refresh(Request()) # OK
```
### Suggested fix
Apply scopes where they are needed, leaving credentials that do not need them untouched:
```python
from google.auth.credentials import with_scopes_if_required
credentials = with_scopes_if_required(
credentials, ["https://www.googleapis.com/auth/cloud-platform"]
)
```
or resolve with `google.auth.default(scopes=[...])`, which routes through the same helper.
Worth noting for whichever is chosen: `with_scopes_if_required` rewrites any credential whose `requires_scopes` is true, and that includes `google.auth.compute_engine.Credentials` — measured `requires_scopes is True` on 2.57.0. On GCE the metadata server ignores the scopes parameter, but on Cloud Run / App Engine Flex it honours it, so this is not a no-op everywhere in-cloud even though it is harmless.
### Workaround
Build the exporters in the application instead of letting `otel_to_cloud=True` resolve ADC, and pass `otel_to_cloud=False` so a second, unscoped set is not installed:
```python
creds, project = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
hooks = get_gcp_exporters(
enable_cloud_tracing=True, enable_cloud_metrics=True, enable_cloud_logging=True,
google_auth=(creds, project),
)
maybe_set_otel_providers([hooks], otel_resource=get_gcp_resource(project))
```
Two things have to be reproduced by hand when doing this, and both are easy to miss:
* `_setup_instrumentation_lib_if_installed` — without `GoogleGenAiSdkInstrumentor` the export carries agent spans and no model calls (plus the HTTPX/gRPC instrumentors on Agent Engine).
* `maybe_install_request_metrics_middleware`, which `get_fast_api_app` skips under `otel_to_cloud=False`. On Agent Engine that is not just a flush-timing loss: `_RequestDrivenMetricReader` has no background ticker, `note_request_start` is the only writer of its in-flight counter, and the generate_content path refuses to collect while that counter is zero — so metrics stop entirely.
Possibly related: discussion #1009 reports the same error with a key file.
### Related, same off-GCP class (observed after applying the workaround above)
Once the token grant succeeds, metric export from a non-GCP host is still rejected by `telemetry.googleapis.com/v1/metrics` with `400 INVALID_ARGUMENT: prometheus_target resource type must have an instance specified`. Off Agent Engine, `get_gcp_resource()` builds a bare `Resource({"gcp.project_id": ...})`; the Telemetry API's Prometheus mapping requires `instance` (from `service.instance.id`) and `location` on the resource, and the GCP resource detector supplies neither outside Google Cloud. Setting `OTEL_RESOURCE_ATTRIBUTES=service.instance.id=,location=` makes the same export return 200, so it is only the resource that is missing. Traces are unaffected. Mentioned here because it hits the same deployment shape (key file, self-hosted) right after the scope problem is fixed; happy to open it separately if preferred.
贡献指南
评估
这个 Issue 还没有评估数据。