googleapis / googleapis/google-cloud-python

(google-cloud-spanner) Built-in metrics fail when one process uses databases in different projects

Open
#18,155 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
3d 4h
Merged PRs (30d)
122

Description

#### Environment details

- OS type and version: Linux (also happens on macOS)
- Python version: 3.13
- `google-cloud-spanner` version: 3.68.0

#### Steps to reproduce

1. In one process, create two clients for databases in two different projects.
2. Run a query on each.
3. Wait a minute for the metrics export.

#### Code example

```python
from google.cloud import spanner

a = spanner.Client(project="project-a")
b = spanner.Client(project="project-b")

a.instance("i").database("d").snapshot().execute_sql("SELECT 1")
b.instance("i").database("d").snapshot().execute_sql("SELECT 1")
```

#### What happens

The metrics export fails, and no metrics show up in Cloud Monitoring for either project:

```
InvalidArgument: 400 Field resource.labels.project_id had an invalid value of
"project-b": if present, must be the project number or ID in the request name
(projects/project-a).
```

Whichever client is created first decides where all metrics go, so everything from the other project is rejected — and it takes the first project's metrics down with it, since they're sent in the same request.

#### What I expected

Metrics for each database go to that database's project. Running against several projects is pretty normal, and right now it means no built-in metrics at all.

#### What I tried

`SPANNER_DISABLE_BUILTIN_METRICS=true` and `Client(disable_builtin_metrics=True)` both work, but only by giving up the metrics. As far as I can tell there's no way to pass in my own exporter or meter provider, so there's no way to keep them. Java has this (googleapis/google-cloud-java#13679) and there's an open request for Go (googleapis/google-cloud-go#11204), but I couldn't find anything similar here.

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied two-project example with google-cloud-spanner 3.68.0 and inspect the built-in metrics path used by spanner.Client. Compare the default behavior with SPANNER_DISABLE_BUILTIN_METRICS and Client(disable_builtin_metrics=True), then trace how the first client's project is reused during export. Done means metrics from each database reach its own project without disabling built-in metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.