GoogleCloudPlatform / GoogleCloudPlatform/cloud-run-mesh
Cloud Service Mesh does not correctly continue W3C traceparent context between services, breaking distributed trace parent-child linkage
- Dominant language
- Go
- Stars
- 35
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
When two Cloud Run services communicate via Cloud Service Mesh (`.mesh.internal`), the resulting Cloud Trace spans do not nest correctly across the service boundary — the downstream service's spans appear as a sibling group under the same trace ID, rather than nested as children of the calling service's outbound request span. The same application code, calling the downstream service directly in an environment without Cloud Service Mesh enabled, produces correctly nested spans.
## Environment
- Cloud Run, Cloud Service Mesh for Cloud Run (Preview)
- Service A: Node.js, OpenTelemetry JS SDK (`@opentelemetry/instrumentation-http`), composite propagator with W3C `traceparent` configured as authoritative over the legacy `x-cloud-trace-context` format (b3 also propagated)
- Service B: Go, OpenTelemetry Go SDK, same propagator priority (traceparent authoritative)
- Service A calls Service B via `http://service-b.mesh.internal`
## Steps to reproduce
1. Deploy two Cloud Run services connected via Cloud Service Mesh, each instrumented with an OpenTelemetry SDK that propagates the W3C `traceparent` header (in our case alongside `x-cloud-trace-context` and b3 headers, for compatibility).
2. Service A makes an outbound HTTP call to Service B via its `.mesh.internal` address.
3. Inspect the resulting trace in Cloud Trace.
## Expected behavior
Service B's spans should nest as children of Service A's outbound-call span, forming a single connected trace tree.
## Actual behavior
Service B's spans appear as a sibling group alongside Service A's spans, both under the same trace ID, but with no parent-child relationship between them — the trace ID is preserved across the mesh hop, but the specific parent span reference is not.
## Additional context
- We've controlled for and ruled out an application-level bug: the exact same application code, deployed to an environment without Cloud Service Mesh (calling the downstream service directly over HTTP), produces correctly nested spans. Re-introducing Cloud Service Mesh reproduces the broken nesting reliably.
- We suspect the mesh's own sidecar-level tracing integration may be generating/continuing trace context based on the legacy `X-Cloud-Trace-Context` header format independently of the `traceparent` header our application-level OpenTelemetry SDKs treat as authoritative (per [Google's own guidance](https://cloud.google.com/trace/docs/trace-context#legacy-http-header) recommending `traceparent` be prioritised over the legacy header). If the sidecar updates one header format's "current span" reference without keeping the other in sync while proxying the request, that would produce exactly this symptom.
- Happy to provide further detail (sanitised trace/header samples, etc.) if it would help investigate — let us know what would be useful.
Contributor guide
Research direction
Start by reproducing the two-service Cloud Run case through the .mesh.internal address and inspect the Cloud Service Mesh sidecar's handling of traceparent and x-cloud-trace-context. Use sanitised trace and header samples to identify where the parent span reference is lost; done means downstream spans nest under the outbound-call span through the mesh as they do in the direct-HTTP control case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, node.js
- Domain
- cloud, distributed-systems, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100