NVIDIA / NVIDIA/nvcf

feat(observability): export uploader telemetry via BYOO OTLP

Open
#1,046 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Why

The request-trace-uploader needs its own operational telemetry through the workload-scoped BYOO Collector. A local Prometheus scrape endpoint is not the uploader telemetry contract.

Scope

  • Replace the uploader-local Prometheus registry and /metrics endpoint with direct, signal-specific OTLP export for metrics.
  • Export uploader structured logs directly through the signal-specific OTLP logs endpoint and protocol.
  • Configure tracing from the signal-specific OTLP traces endpoint and protocol. Use W3C context propagation and bounded server spans for health endpoints, then upload-operation spans as those operations land.
  • Use only OTEL_EXPORTER_OTLP_LOGS_ENDPOINT and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL for logs, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL for metrics, and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL for traces. Support grpc, http, and http/protobuf.
  • Do not use OTEL_EXPORTER_OTLP_ENDPOINT as a fallback. It can describe Collector outbound configuration rather than the workload-to-Collector path.
  • Retain health-only /livez and /readyz endpoints. Rename the listener setting from METRICS_ADDR to HEALTH_ADDR.
  • Emit at minimum these instruments: dynamo.request_trace.uploads with attributes backend and status; dynamo.request_trace.upload.duration with attribute backend; dynamo.request_trace.source_bytes and dynamo.request_trace.transport_bytes with attribute backend; dynamo.request_trace.confirm_latency with attribute backend, measured from segment closure to confirmed terminal success; dynamo.request_trace.records_dropped with attributes record_type and reason; dynamo.request_trace.records_unparseable with attribute reason; dynamo.request_trace.deletions with attribute status; dynamo.request_trace.pending_segments; dynamo.request_trace.quarantined_segments; dynamo.request_trace.quarantine_evicted; dynamo.request_trace.last_success_timestamp with attribute backend; and process uptime.
  • Expose counts of records received per record type so they can be reconciled against Dynamo's own capture counters. A gap caused upstream of the shared volume must be distinguishable from an absence of traffic.
  • Emit spans covering segment read and parse, transform, submission, and confirmation. Because submission and confirmation are independent steps that may occur in different processes, the confirmation span must reference its submission span through a span link rather than a shared attribute, since segment names and remote upload identifiers may not be exported as attributes.
  • Carry the enclosing span's trace and span identifiers on every log record emitted within a span. Severity must distinguish routine progress from quarantine, quarantine eviction, circuit-breaker trips, retry exhaustion, and configuration faults.
  • Write backend error text, endpoints, buckets, filenames, remote upload identifiers, and request identifiers to the local quarantine record on disk. These must not be exported through OTLP logs, traces, or metrics, even though failure diagnosis requires retaining them.
  • Export only finite metric attributes. Do not emit request payloads, credentials, paths, filenames, upload IDs, request IDs, NCA IDs, or error text as metric attributes, trace attributes, or log fields.
  • Flush providers with a bounded timeout on SIGTERM. Export failures must not change readiness.

Tests

  • Direct OTLP delivery tests for logs, traces, and metrics for every supported protocol.
  • Unit tests for endpoint and protocol selection, W3C propagation, structured-log redaction, and bounded trace attributes.
  • Metric-reader tests for every instrument named above, with finite attributes.
  • Span-link tests proving a confirmation span resolves to its submission span across a process restart.
  • Tests proving backend error text and remote upload IDs reach the on-disk quarantine record and never reach any OTLP signal.
  • NVCF integration coverage that proves the uploader receives signal-specific logs, metrics, and traces endpoint plus protocol for its workload-scoped BYOO Collector.

Non-goals

  • A PodMonitor or Prometheus scrape path for uploader metrics.
  • Sending request payload records through OTLP logs.

Parent: #1004

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no source files, so start by locating the request-trace-uploader implementation, its local Prometheus registry and /metrics endpoint, and the METRICS_ADDR setting. Read the existing health and upload paths first; done requires signal-specific OTLP delivery, redaction and span-link tests, bounded shutdown flushing, and NVCF BYOO integration coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.