GoogleCloudPlatform / GoogleCloudPlatform/opentelemetry-cloud-run
Not all traces are getting to GCP
- Dominant language
- Go
- Stars
- 101
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
Original issue: https://github.com/open-telemetry/community/discussions/2082 by @sdsani
Hi Guys,
Our team has been asked to deploy an application in google cloud run as a service. Among many other tasks, one action item is about setting up tracing and monitoring for the application. We decided here to use Open telemetry here. We have two services. One is a ReactJS app while second is a spring boot app. Both are using otel collector as a side car here. Below are the few docs that we have followed here to configure our application.
https://cloud.google.com/run/docs/tutorials/custom-metrics-opentelemetry-sidecar
https://opentelemetry.io/docs/languages/java/automatic/spring-boot/
And many other docs.
We got it working at the end. We can see traces, showing up in google cloud traces. Metrics are getting into metrics and so on.
However, during testing, we found that not all traces are getting into the google cloud trace. Sometime every second trace get into cloud trace, while sometime every fourth get into the cloud trace. We have looked many different settings and played around that, however, none of these settings appears to make any difference. To get this capability, we need a more reliable config in place since for production support, missing these traces would not be an option.
Below is the image that we are using for otel collector sidecar (pulled from dockerhub).
otel/opentelemetry-collector-contrib:0.99.0
This image has been customized with a config file and attached is that config file (collector-config.yaml.txt)
Attached is also yaml file for cloud run service config (cloud-run-service-config.yaml.txt)
Please advise.
[cloud-run-service-config.yaml.txt](https://github.com/open-telemetry/community/files/15165363/cloud-run-service-config.yaml.txt)
[collector-config.yaml.txt](https://github.com/open-telemetry/community/files/15165364/collector-config.yaml.txt)
In our spring boot config, our team is already setting sampling rate to 1.0.
Have tried adding following to the env for the collector sidecar also and this is not helping either.
env:
- name: OTEL_TRACES_SAMPLER
value: traceidratio
- name: OTEL_TRACES_SAMPLER_ARG
value: "1.0"
Contributor guide
Research direction
Start by reviewing the attached collector-config.yaml.txt and cloud-run-service-config.yaml.txt alongside the Cloud Run sidecar and Spring Boot tracing setup described in the issue. Reproduce or analyze the intermittent export behavior and verify whether the configuration reliably delivers all sampled traces to Google Cloud Trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, react, spring-boot
- Domain
- cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100