prometheus-community / prometheus-community/stackdriver_exporter
Errors when specifying multiple projects and spanner.googleapis.com
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 295
- Forks
- 117
- Avg merge
- 9d 14h
- Merged PRs (30d)
- 7
Description
Version 0.18.0
When specifying multiple projects
/bin/stackdriver_exporter \
--google.project-id=gitlab-runway-staging,gitlab-runway-topo-svc-stg \
--monitoring.metrics-prefixes=spanner.googleapis.com \
--web.listen-address=:9256 \
--web.telemetry-path=/metrics
* [from Gatherer #2] collected metric "stackdriver_spanner_instance_spanner_googleapis_com_client_gfe_connectivity_error_count" { label:{name:"client_name" value:"spanner-go/1.83.0"} label:{name:"database" value
:"topology-service"} label:{name:"directpath_enabled" value:"false"} label:{name:"directpath_used" value:"false"} label:{name:"instance_config" value:"unknown"} label:{name:"instance_id" value:"topology-ser
vice-dev"} label:{name:"location" value:"asia-southeast1"} label:{name:"method" value:"Spanner.ExecuteSql"} label:{name:"project_id" value:"gitlab-runway-topo-svc-stg"} label:{name:"status" value:"Unavailab
le"} label:{name:"unit" value:"1"} gauge:{value:0} timestamp_ms:1763391420000} was collected before with the same name and label values
All errors are from Gather #2 so it looks like there may be a race condition when you don't specify individual metric prefixes.
I can work around this with:
/bin/stackdriver_exporter \
--google.project-id=gitlab-runway-staging,gitlab-runway-topo-svc-stg \
--monitoring.metrics-prefixes=spanner.googleapis.com/instance/cpu \
--monitoring.metrics-prefixes=spanner.googleapis.com/instance/storage \
--monitoring.metrics-prefixes=spanner.googleapis.com/instance/sessions \
--monitoring.metrics-prefixes=spanner.googleapis.com/api/request \
--monitoring.metrics-prefixes=spanner.googleapis.com/lock_stat \
--web.listen-address=:9256 \
--web.telemetry-path=/metrics
And then I don't see the error.
This issue only manifests with multiple projects using the --google.project-id option
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the shown stackdriver_exporter command, using multiple --google.project-id values and the broad spanner.googleapis.com prefix. Start by tracing the Gather #2 path and compare it with the individual-prefix workaround. Done means multiple projects can be scraped without duplicate metric collection errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100