NVIDIA / NVIDIA/nvcf

feat(request-trace-uploader): parse v1.4.0 records and add initial object-store upload

Open
#1,047 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Why

The uploader scaffold only discovers closed Dynamo request-trace segments. The next increment should prove the narrow happy path, and pin the record format it depends on, before adding durable recovery and configurable policy complexity.

Scope

  • Parse the Dynamo v1.4.0 RequestTraceRecord format. All five event types (request_end, request_payload, tool_start, tool_end, tool_error) must parse, including records where the optional request, payload, and tool members are absent.
  • Resolve the request identifier from request.request_id on request_end and payload.request_id on request_payload. Parse both the bare record form and the timestamp-wrapped form.
  • Detect the Dynamo v1.3.x AuditRecord shape and fail with an explicit version error rather than mis-parsing or silently discarding records.
  • Export records carrying an unrecognized event type or unrecognized fields with content preserved verbatim, counted rather than dropped.
  • Isolate parse failures per record. A record that fails to parse must not fail the segment containing it: skip it, preserve its raw bytes for inspection, count it, and continue. Quarantine the segment only when a configured proportion of its records fail.
  • Classify records by event_type only. Dynamo v1.4.0 writes a single file family; there is no separate audit directory or prefix.
  • Define the backend interface with separate submit and confirm steps, and a capability declaration covering resubmit safety, terminal-outcome timing, out-of-order tolerance, accepted formats, and maximum object size.
  • Implement the generic S3-compatible object-store backend against that interface.
  • Read the next closed segment, transform it into the format the selected backend declares it accepts, submit it, and delete the source only after a successful upload response.
  • On a failed read, parse, transform, upload, or delete, retain the source file and return the error through structured logs and telemetry.
  • Process one closed segment at a time. Do not add background queues, retries, persistent state, status polling, quarantine, or automatic deployment injection in this increment.

Tests

  • Closed-versus-active segment selection.
  • Every event type, including absent optional members, and both record forms.
  • Fixtures captured from a running Dynamo instance at the pinned version, with the producing version recorded alongside them.
  • v1.3.x AuditRecord input produces a version error.
  • Unknown event type and unknown field preservation.
  • Successful read, transform, upload, and source deletion.
  • Read, transform, upload, and delete failure paths that retain the source and produce bounded telemetry.
  • Concatenated gzip input and destination-envelope tests.

Follow-on work:

  • Durable lifecycle state and failure quarantine.
  • Configurable upload policies.
  • Kratos Bulk Upload backend.
  • Record suppression and sampling.
  • Full observability and documentation.

Parent: #1004

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the request-trace-uploader scaffold, closed-segment reader, and existing tests for segment selection and record parsing. Done means v1.4.0 records and failure paths are covered, the S3-compatible backend submits and confirms through the declared interface, and a source segment is deleted only after successful upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.