feat(observability): export uploader telemetry via BYOO OTLP
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Why
The request-trace-uploader needs its own operational telemetry through the workload-scoped BYOO Collector. A local Prometheus scrape endpoint is not the uploader telemetry contract.
Scope
- Replace the uploader-local Prometheus registry and
/metricsendpoint with direct, signal-specific OTLP export for metrics. - Export uploader structured logs directly through the signal-specific OTLP logs endpoint and protocol.
- Configure tracing from the signal-specific OTLP traces endpoint and protocol. Use W3C context propagation and bounded server spans for health endpoints, then upload-operation spans as those operations land.
- Use only
OTEL_EXPORTER_OTLP_LOGS_ENDPOINTandOTEL_EXPORTER_OTLP_LOGS_PROTOCOLfor logs,OTEL_EXPORTER_OTLP_METRICS_ENDPOINTandOTEL_EXPORTER_OTLP_METRICS_PROTOCOLfor metrics, andOTEL_EXPORTER_OTLP_TRACES_ENDPOINTandOTEL_EXPORTER_OTLP_TRACES_PROTOCOLfor traces. Supportgrpc,http, andhttp/protobuf. - Do not use
OTEL_EXPORTER_OTLP_ENDPOINTas a fallback. It can describe Collector outbound configuration rather than the workload-to-Collector path. - Retain health-only
/livezand/readyzendpoints. Rename the listener setting fromMETRICS_ADDRtoHEALTH_ADDR. - Emit at minimum these instruments:
dynamo.request_trace.uploadswith attributes backend and status;dynamo.request_trace.upload.durationwith attribute backend;dynamo.request_trace.source_bytesanddynamo.request_trace.transport_byteswith attribute backend;dynamo.request_trace.confirm_latencywith attribute backend, measured from segment closure to confirmed terminal success;dynamo.request_trace.records_droppedwith attributes record_type and reason;dynamo.request_trace.records_unparseablewith attribute reason;dynamo.request_trace.deletionswith attribute status;dynamo.request_trace.pending_segments;dynamo.request_trace.quarantined_segments;dynamo.request_trace.quarantine_evicted;dynamo.request_trace.last_success_timestampwith attribute backend; and process uptime. - Expose counts of records received per record type so they can be reconciled against Dynamo's own capture counters. A gap caused upstream of the shared volume must be distinguishable from an absence of traffic.
- Emit spans covering segment read and parse, transform, submission, and confirmation. Because submission and confirmation are independent steps that may occur in different processes, the confirmation span must reference its submission span through a span link rather than a shared attribute, since segment names and remote upload identifiers may not be exported as attributes.
- Carry the enclosing span's trace and span identifiers on every log record emitted within a span. Severity must distinguish routine progress from quarantine, quarantine eviction, circuit-breaker trips, retry exhaustion, and configuration faults.
- Write backend error text, endpoints, buckets, filenames, remote upload identifiers, and request identifiers to the local quarantine record on disk. These must not be exported through OTLP logs, traces, or metrics, even though failure diagnosis requires retaining them.
- Export only finite metric attributes. Do not emit request payloads, credentials, paths, filenames, upload IDs, request IDs, NCA IDs, or error text as metric attributes, trace attributes, or log fields.
- Flush providers with a bounded timeout on SIGTERM. Export failures must not change readiness.
Tests
- Direct OTLP delivery tests for logs, traces, and metrics for every supported protocol.
- Unit tests for endpoint and protocol selection, W3C propagation, structured-log redaction, and bounded trace attributes.
- Metric-reader tests for every instrument named above, with finite attributes.
- Span-link tests proving a confirmation span resolves to its submission span across a process restart.
- Tests proving backend error text and remote upload IDs reach the on-disk quarantine record and never reach any OTLP signal.
- NVCF integration coverage that proves the uploader receives signal-specific logs, metrics, and traces endpoint plus protocol for its workload-scoped BYOO Collector.
Non-goals
- A PodMonitor or Prometheus scrape path for uploader metrics.
- Sending request payload records through OTLP logs.
Parent: #1004
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The payload names no source files, so start by locating the request-trace-uploader implementation, its local Prometheus registry and /metrics endpoint, and the METRICS_ADDR setting. Read the existing health and upload paths first; done requires signal-specific OTLP delivery, redaction and span-link tests, bounded shutdown flushing, and NVCF BYOO integration coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100