GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine
Easier Cloud Trace integration by automating project_id requirements
- Dominant language
- Go
- Stars
- 232
- Forks
- 109
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 27
Description
_Disclaimer: I am unsure if this is the correct place for the question or if there are already existing solutions._
TLDR: Looking at the [docs](https://cloud.google.com/stackdriver/docs/managed-prometheus/exemplars#trace-compatibility), the requirements for GCM - Cloud Trace integration include having `project_id` as a label on the exemplar itself. This is not always an easy requirement, and having GCM add that automatically would be easier from a consumer point of view.
The following is a detailed description of the problem.
## Using a custom application with OpenTelemetry metrics
I have a Go application on GKE which:
* Uses an OpenTelemetry TracerProvider exporting traces to a Google-Build OpenTelemetry Collector ([docs](https://cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-gke)).
* Uses an OpenTelemetry MeterProvider exporting metrics via a Prometheus HTTP server and Google Cloud Managed Service for Prometheus scraping the endpoint using a PodMonitoring CR.
Now I am looking into the Google Cloud Managed Service for Prometheus integration with Cloud Trace as described [here](https://cloud.google.com/stackdriver/docs/managed-prometheus/exemplars#trace-compatibility). The exemplars in Cloud Monitoring, which I received using a ListTimeSeries RPC call look like this (for a metric `prometheus/grpc_server_call_duration_seconds/histogram`):
```
exemplars": [
{
"value": 0.00946708,
"timestamp": "2025-09-11T08:01:22.777Z",
"attachments": [
{
"@type": "type.googleapis.com/google.monitoring.v3.DroppedLabels",
"label": {
"resource.labels.namespace": "",
"resource.labels.location": "europe-west1-b",
"resource.labels.cluster": "development",
"resource.labels.job": "",
"project_id": "",
"resource.labels.instance": ""
}
}
]
}
]
```
Note that there is no SpanContext created. If I directly scrape the Prometheus endpoint using GKE port forwarding and `curl -H "Accept: application/openmetrics-text" http://localhost:8080/metrics`, I do see the trace_id and span_id showing up but no project_id (as expected).
A workaround is probably to
* Use the OpenTelemetry collector for metrics as well. This option I would like to avoid.
* Add a custom wrapping OpenTelemetry Gatherer which adds the project_id label on each exemplar. However it would require quite some application changes.
## Using go-synthetic using Prometheus directly
Testing the go-synthetic example from this repository (which uses the Prometheus client directly), I do see the SpanContext created. This example correctly sets the project ID as an exemplar label [here](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/examples/instrumentation/go-synthetic/main.go#L437) (it's of course not the correct project ID).
```
"exemplars": [
{
"value": -229.4003856706737,
"timestamp": "2025-09-11T10:01:33.719Z",
"attachments": [
{
"@type": "type.googleapis.com/google.monitoring.v3.SpanContext",
"spanName": "projects/example-project/traces/c8e03aa73b21f6bb13f2716f179a6fa8/spans/838d17d28b312737"
},
{
"@type": "type.googleapis.com/google.monitoring.v3.DroppedLabels",
"label": {
"resource.labels.location": "europe-west1-b",
"resource.labels.job": "go-synthetic",
"project_id": "",
"resource.labels.instance": "",
"resource.labels.cluster": "development",
"resource.labels.namespace": ""
}
}
]
}
]
```
Contributor guide
Assessment
This issue has not been assessed yet.