googleapis / googleapis/google-cloud-python

InvalidArgument error: "resource labels is incomplete, missing (instance_id)" when exporting built-in metrics

Đang mở
#15,879 3 bình luận 3 reaction 1 người được giao Được giao cho @sinhasubham Xem trên GitHub
api: spanner priority: p1 type: process
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

After upgrading to `google-cloud-spanner` v3.60.0+, the built-in OpenTelemetry metrics feature (enabled by default) fails to export metrics to Cloud Monitoring with the following error:

```text
InvalidArgument: One or more TimeSeries could not be written:
timeSeries[N] (metric.type="spanner.googleapis.com/internal/client/operation_latencies", ...):
the set of resource labels is incomplete, missing (instance_id)
```

**Impact**: This error does not affect Spanner operations (queries succeed with HTTP 200), but it:

- Generates significant log noise
- Floods error tracking systems (e.g., Sentry) with thousands of errors
- Prevents built-in metrics from being collected in Cloud Monitoring

---

## Environment

- **google-cloud-spanner version**: 3.62.0 (also affects 3.60.0, 3.61.0)
- **Python version**: 3.13.8
- **OS**: Linux (Cloud Run)
- **Previous working version**: 3.59.x and earlier (metrics were opt-in)

---

## Minimal Reproduction

```python
from google.cloud import spanner

# With google-cloud-spanner >= 3.60.0, built-in metrics are enabled by default
client = spanner.Client(project="my-project")
instance = client.instance("my-instance")
database = instance.database("my-database")

with database.snapshot() as snapshot:
results = snapshot.execute_sql("SELECT 1")
list(results) # Execute query

# After ~60 seconds (metrics export interval), the following error appears in logs:
# google.api_core.exceptions.InvalidArgument: ... missing (instance_id)
```

---

## Expected Behavior

All required resource labels, including `instance_id`, are set, and the error does not occur.

---

## Actual Behavior

Metrics export fails with `InvalidArgument` error. The resource labels in the error message show:

```text
resource.labels={"location": "asia-northeast1", "instance_config": "unknown", "client_hash": "XXXXXX"}
```

## Stack Trace

```text
File "opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
self._exporter.export(
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 357, in export
self._batch_write(time_series_list, timeout_millis)
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 154, in _batch_write
self.client.create_service_time_series(
File "google/cloud/monitoring_v3/services/metric_service/client.py", line 1937, in create_service_time_series
rpc(
File "google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: One or more TimeSeries could not be written:
... the set of resource labels is incomplete, missing (instance_id)
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.