googleapis / googleapis/google-cloud-python

InvalidArgument error: "resource labels is incomplete, missing (instance_id)" when exporting built-in metrics

Abierto
#15,879 3 comentarios 3 reacciones 1 asignado Asignado a @sinhasubham Ver en GitHub
api: spanner priority: p1 type: process
Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
3 d 4 h
PR fusionados (30 d)
122

Descripción

After upgrading to `google-cloud-spanner` v3.60.0+, the built-in OpenTelemetry metrics feature (enabled by default) fails to export metrics to Cloud Monitoring with the following error:

```text
InvalidArgument: One or more TimeSeries could not be written:
timeSeries[N] (metric.type="spanner.googleapis.com/internal/client/operation_latencies", ...):
the set of resource labels is incomplete, missing (instance_id)
```

**Impact**: This error does not affect Spanner operations (queries succeed with HTTP 200), but it:

- Generates significant log noise
- Floods error tracking systems (e.g., Sentry) with thousands of errors
- Prevents built-in metrics from being collected in Cloud Monitoring

---

## Environment

- **google-cloud-spanner version**: 3.62.0 (also affects 3.60.0, 3.61.0)
- **Python version**: 3.13.8
- **OS**: Linux (Cloud Run)
- **Previous working version**: 3.59.x and earlier (metrics were opt-in)

---

## Minimal Reproduction

```python
from google.cloud import spanner

# With google-cloud-spanner >= 3.60.0, built-in metrics are enabled by default
client = spanner.Client(project="my-project")
instance = client.instance("my-instance")
database = instance.database("my-database")

with database.snapshot() as snapshot:
results = snapshot.execute_sql("SELECT 1")
list(results) # Execute query

# After ~60 seconds (metrics export interval), the following error appears in logs:
# google.api_core.exceptions.InvalidArgument: ... missing (instance_id)
```

---

## Expected Behavior

All required resource labels, including `instance_id`, are set, and the error does not occur.

---

## Actual Behavior

Metrics export fails with `InvalidArgument` error. The resource labels in the error message show:

```text
resource.labels={"location": "asia-northeast1", "instance_config": "unknown", "client_hash": "XXXXXX"}
```

## Stack Trace

```text
File "opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
self._exporter.export(
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 357, in export
self._batch_write(time_series_list, timeout_millis)
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 154, in _batch_write
self.client.create_service_time_series(
File "google/cloud/monitoring_v3/services/metric_service/client.py", line 1937, in create_service_time_series
rpc(
File "google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: One or more TimeSeries could not be written:
... the set of resource labels is incomplete, missing (instance_id)
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.