googleapis / googleapis/google-cloud-go
firestore: trace span names contain high cardinality ids
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
## Client
Firestore sets trace span names with id in them, resulting in polluting traces with garbage and high cardinality and cost. you would get something like /my_collection/_doc/asdf1234 traces of kind client
this is very bad, I just found out our spants are polluted with this garbage produced by official GCP Go Firestore client. this is so bad, I have to shutdown whole firestore telemetry, because it is a garbage firehose into our telemetry.
trace spans `MUST NOT` contain high cardinality IDs. the purpose of span is to **group** repeated invocations to track their: latency; error rate; volume of requests. now, if we have unique IDs there.. this does not make sense, likelihood of hitting same ID is very low and it is not useful.... but whom I am explaning this all about? you guys written "Site Reliability at Google" and pionireed SRE, right?
please fix this!
## Environment
- Alpine Docker on GKE
- go 1.25
## Code and Dependencies
we are using OpenTelemetry and GCP Cloud Monitoring as a sink for traces. Grafana dashbaord based on span_metrics OTEL Plugin: https://github.com/ndx-technologies/grafana-open-telemetry-apm + https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector
go.mod
```
cloud.google.com/go/firestore v1.18.0
...
go.opentelemetry.io/contrib/detectors/gcp v1.38.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
go.opentelemetry.io/contrib/instrumentation/runtime v0.63.0
go.opentelemetry.io/otel v1.38.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.38.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0
go.opentelemetry.io/otel/metric v1.38.0
go.opentelemetry.io/otel/sdk v1.38.0
go.opentelemetry.io/otel/sdk/metric v1.38.0
go.opentelemetry.io/otel/trace v1.38.0
```
## Expected behavior
no high cardniality IDs in trace span names. only in labels.
## Actual behavior
high cardinality IDs in trace span names.
## Screenshots
## Additional context
N/A
Contributor guide
Assessment
This issue has not been assessed yet.