Add OTLP Trace Support for Remote Execution Observability

Problem Statement

Currently, NativeLink lacks comprehensive observability into the distributed build execution pipeline. When tasks experience delays or failures, users cannot easily determine:

• Which worker node is executing a specific task
• Current execution state (queued, running, completed)
• Detailed execution logs and command content
• Complete call chain from Bazel client to worker execution

While the system provides some task dispatch information, there is no complete tracing capability, especially for retrieving actual command content from command_digest, making troubleshooting inefficient.

Proposal

I propose integrating OpenTelemetry (OTLP) tracing into NativeLink to provide end-to-end visibility of build task execution.

Core Features

1. Use Bazel Invocation ID as Trace ID

Leverage tool_invocation_id from Bazel invocation as the root trace identifier to enable correlation of all operations within a single Bazel build.

2. Hierarchical Span Structure

TraceID (tool_invocation_id)
└── action_mnemonic, action_digest
└── client_operation_id
└── operation_id
├── Command Span
├── Prepare Span
└── Execution Span
└── exit_code

3. Key Span Attributes

• action_id (action_digest): Action identifier
• action_mnemonic: Operation type (CppCompile, CppLink, TestRunner, etc.)
• target_id: Bazel target (e.g., //aios/alog:alog)
• client_operation_id: User-side operation ID
• operation_id: Worker execution ID
• command_digest: Task execution command
• worker_id: Worker node identifier
• exit_code: Execution result

4. Multi-dimensional Query Support

• Search by Bazel invocation ID
• Filter by action mnemonic
• Locate by target ID
• Match by command or output files

Technical Approach

Context Propagation

The key challenge is passing trace context through:
Bazel Client → Scheduler → Redis → Scheduler → Worker

Solution:

1. Create trace context when Scheduler receives request
2. Propagate trace_id and span_id through Redis ActionInfo using Baggage or custom fields
3. Worker restores trace context from task data

Span Lifecycle

1. Scheduler receives request: Create root span with Bazel invocation ID
2. Action queued: Record queueing span with task requirements
3. Worker pulls task: Create execution span with worker_id
4. Command execution: Record command, environment, and duration
5. Task completion: Record exit code and output

Benefits

• Faster troubleshooting: Quickly identify which worker is handling a task
• Complete visibility: See the entire execution pipeline in one view
• Standard tooling: Compatible with Jaeger, Grafana Tempo, and other OTLP-compatible backends
• Flexible querying: Search by multiple dimensions (invocation ID, mnemonic, target)

Integration Example

Users could query traces using:

# Based on Bazel output
Invocation ID: 54d06053-da11-400e-8cf1-965227955a28

# In trace UI (Jaeger/Grafana)
- Search by trace_id: 54d06053-da11-400e-8cf1-965227955a28
- Filter spans by attributes:
  - action_mnemonic: "CppCompile"
  - target_id: "//aios/alog:AppenderTest"
  - command: contains "foo.cc"

Next Steps
If the maintainers believe this feature would be valuable for NativeLink, I would be happy to provide my implementation. I have already developed a working prototype that includes:

• Basic OTLP infrastructure integration
• Trace context propagation through Redis/MemoryStore and through worker_api with proto field added
• Key span instrumentation at critical points
• Sample configuration for common OTLP backends (Jaeger)
  Looking forward to your feedback on whether this aligns with NativeLink's roadmap and architecture principles!

No repository files or tests are named. Start by reviewing the scheduler, Redis/MemoryStore, and worker_api areas described in the proposal, then confirm the maintainers' intended scope; done would require an agreed design and end-to-end OTLP trace coverage across the execution pipeline.