feat(compute-plane): add request-trace-uploader image

Why

Dynamo request tracing writes closed, compressed RequestTraceRecord segments to a shared volume. Input and output payload capture uses the request_payload record type. These records are neither normal container logs nor distributed traces.

This umbrella tracks building the focused request-trace-uploader image. The image owns only the closed Dynamo request-trace segment to object-store lifecycle.

Scope

The uploader is not a good fit for an OTel Collector deployment. The optimal solution is a focused binary that does this one job of exporting Dynamo request trace records to an object store.

Dynamo v1.4.0 is the minimum supported version. It writes all record types to a single rolling file family, so record classification comes from event_type on each record, never from a filename or a separate audit directory. The five event types are request_end, request_payload, tool_start, tool_end, and tool_error. Dynamo v1.3.x emits a separate AuditRecord type that this uploader does not parse and must reject with an explicit version error.

Export goes through a pluggable backend interface with two implementations: a generic S3-compatible object store, and Kratos Bulk Upload. Backends declare their own capabilities — resubmit safety, terminal-outcome timing, out-of-order tolerance, accepted formats, maximum object size — and core behavior derives from those declarations rather than branching on backend identity. The Kratos implementation sits behind a build tag so the core and object-store path compile without the internal Kratos SDK.

The uploader preserves a source until durable terminal success, and never logs credentials or request payloads.

Delivery plan

1. #1041: bare-bones scaffold. Build and test the image with configuration validation, closed-segment discovery, and health only. Do not publish, read, upload, delete, or expose metrics in this increment.
2. #1047: parse the v1.4.0 record format, read one closed segment, upload it to an object store, and delete it only after success. Includes the backend interface and capability declaration.
3. #1050: add durable lifecycle state, restart recovery, fault scoping, and bounded failure quarantine.
4. #1051: add configurable retry, timeout, backoff, size, concurrency, outbox, and asynchronous-status policy.
5. #1443: add the Kratos Bulk Upload backend behind the same interface and a build tag.
6. #1048: add configurable record suppression by header group, applied to request_payload records.
7. #1444: add configurable per-record-type sampling.
8. #1046: add the uploader's own OTLP logs, traces, and metrics.
9. #1052: document deployment, operations, and alerts.

Future first-class deployment is tracked separately in #1049. NVCA must inject the sidecar image, shared and state volumes, credentials, probes, configuration, and BYOO signal configuration. Environment injection alone is not enough.

Producer contract

The uploader cannot observe Dynamo's configuration. Deployments must set DYN_REQUEST_TRACE_RECORDS to include request_payload for payload capture, and DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST to include every header a suppression rule depends on, because Dynamo never reads a header that is not on that allowlist. Deprecated DYN_AUDIT_* variables must not be set: DYN_AUDIT_FORCE_LOGGING silently enables payload capture for every request on v1.4.0.

Telemetry contract

When telemetry is added, the uploader exports its own structured logs, traces, and metrics directly to the workload-scoped BYOO Collector using signal-specific OTLP endpoint and protocol configuration. It must not expose or rely on a Prometheus scrape route, use a generic OTLP endpoint fallback, or send request-record payloads through OTLP logs.

Non-goals

Migrating existing shell-sidecar deployments is out of scope. The shell uploader is expected to be updated independently to work with Dynamo v1.4. The uploader will never encounter shell-uploader state, because that state lives on a pod-local emptyDir that does not survive pod replacement.

Definition of done

• A versioned, tested image is deployable once the upload lifecycle is complete.
• Records of all five v1.4.0 event types round-trip, including records where the optional request, payload, and tool members are absent.
• A deployment pointed at Dynamo v1.3.x fails with an explicit version error rather than mis-parsing records.
• The same image exports to a generic object store with no NVIDIA-internal dependencies, and to Kratos Bulk Upload by changing only the backend setting.
• Source segments are deleted only after durable terminal success. Backend-wide faults retain segments for retry; only segment-specific faults quarantine, and quarantine is bounded.
• Record suppression protects matching request records without removing correlation evidence, and its coverage is observable.
• Uploader logs, traces, and metrics reach BYOO through OTLP.
• The deployment and operational contract is documented.