googleapis / googleapis/google-cloud-python

InvalidArgument error: "resource labels is incomplete, missing (instance_id)" when exporting built-in metrics

Aperta
#15,879 3 commenti 3 reazioni 1 assegnatario Assegnata a @sinhasubham Vedi su GitHub
api: spanner priority: p1 type: process
Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
3g 4h
PR unite (30g)
122

Descrizione

After upgrading to `google-cloud-spanner` v3.60.0+, the built-in OpenTelemetry metrics feature (enabled by default) fails to export metrics to Cloud Monitoring with the following error:

```text
InvalidArgument: One or more TimeSeries could not be written:
timeSeries[N] (metric.type="spanner.googleapis.com/internal/client/operation_latencies", ...):
the set of resource labels is incomplete, missing (instance_id)
```

**Impact**: This error does not affect Spanner operations (queries succeed with HTTP 200), but it:

- Generates significant log noise
- Floods error tracking systems (e.g., Sentry) with thousands of errors
- Prevents built-in metrics from being collected in Cloud Monitoring

---

## Environment

- **google-cloud-spanner version**: 3.62.0 (also affects 3.60.0, 3.61.0)
- **Python version**: 3.13.8
- **OS**: Linux (Cloud Run)
- **Previous working version**: 3.59.x and earlier (metrics were opt-in)

---

## Minimal Reproduction

```python
from google.cloud import spanner

# With google-cloud-spanner >= 3.60.0, built-in metrics are enabled by default
client = spanner.Client(project="my-project")
instance = client.instance("my-instance")
database = instance.database("my-database")

with database.snapshot() as snapshot:
results = snapshot.execute_sql("SELECT 1")
list(results) # Execute query

# After ~60 seconds (metrics export interval), the following error appears in logs:
# google.api_core.exceptions.InvalidArgument: ... missing (instance_id)
```

---

## Expected Behavior

All required resource labels, including `instance_id`, are set, and the error does not occur.

---

## Actual Behavior

Metrics export fails with `InvalidArgument` error. The resource labels in the error message show:

```text
resource.labels={"location": "asia-northeast1", "instance_config": "unknown", "client_hash": "XXXXXX"}
```

## Stack Trace

```text
File "opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
self._exporter.export(
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 357, in export
self._batch_write(time_series_list, timeout_millis)
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 154, in _batch_write
self.client.create_service_time_series(
File "google/cloud/monitoring_v3/services/metric_service/client.py", line 1937, in create_service_time_series
rpc(
File "google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: One or more TimeSeries could not be written:
... the set of resource labels is incomplete, missing (instance_id)
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.