NVIDIA / NVIDIA/OpenShell

feat(cli): passive TRACEPARENT forwarding for CI trace continuity

Open
#2,639 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:accepted topic:observability
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 7h
Merged PRs (30d)
243

Description

Problem Statement

CI pipelines that invoke openshell sandbox create lose trace continuity at the CLI boundary. When a CI system sets TRACEPARENT in the environment (GitHub Actions, GitLab CI, Jenkins with OpenTelemetry (OTel) plugins all do this), the pipeline trace ends at the CLI invocation and a separate, disconnected trace starts at the gateway. The engineer cannot see how long sandbox creation took, which driver handled it, or whether there were retries as part of their pipeline trace.

The gateway already traces every incoming request and exports spans via the OpenTelemetry Protocol (OTLP) (#2534). The missing piece is forwarding the CI-provided trace context from the CLI to the gateway so the gateway's sandbox.create span becomes a child of the pipeline trace.

Proposed Design

Add passive TRACEPARENT forwarding to the CLI's gRPC channel. "Passive" means no SdkTracerProvider, no OTLP export, no collector configuration on the CLI side. The CLI runs on developer laptops, CI runners, and jump hosts where an OTel collector is typically not reachable, and the gateway is already the instrumentation boundary for platform tracing.

The implementation uses TraceContextInterceptor from the openshell-otel crate (#2567), which already exists for the VM driver's gRPC channel. On the CLI's gRPC channel to the gateway:

  1. Read TRACEPARENT (and optionally TRACESTATE) from the process environment
  2. Inject these as gRPC metadata headers via TraceContextInterceptor
  3. The gateway's existing span extraction picks them up and parents its spans accordingly

If TRACEPARENT is not set, the interceptor is a no-op.

Persona workflow: CI engineer tracing a pipeline step through the gateway

A CI pipeline runs openshell sandbox create, executes an agent task, and tears down the sandbox. The pipeline has its own OTel tracing (the CI system sets TRACEPARENT in the environment). The engineer wants to see the OpenShell operations as part of the pipeline trace: how long did sandbox creation take, which driver handled it, were there retries? Today, the CLI does not forward TRACEPARENT to the gateway, so the pipeline trace ends at the CLI invocation and a separate, disconnected trace starts at the gateway.

How this issue enables the workflow: The CLI forwards the TRACEPARENT from the environment to the gateway via TraceContextInterceptor on the gRPC channel. The gateway's sandbox.create span becomes a child of the pipeline trace. The engineer sees the full flow in their CI tracing dashboard: pipeline step -> CLI -> gateway -> K8s driver -> pod provisioning. No OTel provider or collector configuration is needed on the CI runner; the CLI just passes through what the CI system already provides.

Scope
  • crates/openshell-cli/: Add TraceContextInterceptor to the gRPC channel construction
  • No new dependencies (the crate already exists in the workspace)
  • No configuration surface needed (reads from standard TRACEPARENT env var)

Alternatives Considered

Full OTel provider on the CLI: Rejected. This would require every CLI installation to configure collector connectivity. The CLI is not a long-running service; it makes a few gRPC calls and exits. The gateway already traces the server side of every request, so CLI-side spans would be redundant. Passive forwarding gets the benefit (trace continuity) without the operational cost (collector setup everywhere).

Custom header instead of W3C traceparent: Rejected. TRACEPARENT is the W3C standard and is what CI systems set. Using a non-standard header would require CI systems to add OpenShell-specific configuration.

Agent Investigation

  • TraceContextInterceptor already exists in crates/openshell-otel/ and is used on the VM driver's gRPC channel (#2567)
  • The CLI constructs its gRPC channel in crates/openshell-cli/ (client connection setup)
  • The gateway's inbound span extraction from gRPC metadata is already working (#2534)
  • No new crate dependencies needed; openshell-otel is already in the workspace

Related: #1055 (Enterprise Observability), #2507 (Gateway OTel export surface)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect the client connection setup in crates/openshell-cli/ and compare it with the VM driver's gRPC channel use of TraceContextInterceptor from crates/openshell-otel/. Done means the CLI forwards TRACEPARENT and optional TRACESTATE to the gateway, remains a no-op when TRACEPARENT is absent, and adds no new dependencies or CLI configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
cli, distributed-systems, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.