feat(request-trace-uploader): parse v1.4.0 records and add initial object-store upload
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Why
The uploader scaffold only discovers closed Dynamo request-trace segments. The next increment should prove the narrow happy path, and pin the record format it depends on, before adding durable recovery and configurable policy complexity.
Scope
- Parse the Dynamo v1.4.0
RequestTraceRecordformat. All five event types (request_end,request_payload,tool_start,tool_end,tool_error) must parse, including records where the optionalrequest,payload, andtoolmembers are absent. - Resolve the request identifier from
request.request_idonrequest_endandpayload.request_idonrequest_payload. Parse both the bare record form and the timestamp-wrapped form. - Detect the Dynamo v1.3.x
AuditRecordshape and fail with an explicit version error rather than mis-parsing or silently discarding records. - Export records carrying an unrecognized event type or unrecognized fields with content preserved verbatim, counted rather than dropped.
- Isolate parse failures per record. A record that fails to parse must not fail the segment containing it: skip it, preserve its raw bytes for inspection, count it, and continue. Quarantine the segment only when a configured proportion of its records fail.
- Classify records by
event_typeonly. Dynamo v1.4.0 writes a single file family; there is no separate audit directory or prefix. - Define the backend interface with separate submit and confirm steps, and a capability declaration covering resubmit safety, terminal-outcome timing, out-of-order tolerance, accepted formats, and maximum object size.
- Implement the generic S3-compatible object-store backend against that interface.
- Read the next closed segment, transform it into the format the selected backend declares it accepts, submit it, and delete the source only after a successful upload response.
- On a failed read, parse, transform, upload, or delete, retain the source file and return the error through structured logs and telemetry.
- Process one closed segment at a time. Do not add background queues, retries, persistent state, status polling, quarantine, or automatic deployment injection in this increment.
Tests
- Closed-versus-active segment selection.
- Every event type, including absent optional members, and both record forms.
- Fixtures captured from a running Dynamo instance at the pinned version, with the producing version recorded alongside them.
- v1.3.x
AuditRecordinput produces a version error. - Unknown event type and unknown field preservation.
- Successful read, transform, upload, and source deletion.
- Read, transform, upload, and delete failure paths that retain the source and produce bounded telemetry.
- Concatenated gzip input and destination-envelope tests.
Follow-on work:
- Durable lifecycle state and failure quarantine.
- Configurable upload policies.
- Kratos Bulk Upload backend.
- Record suppression and sampling.
- Full observability and documentation.
Parent: #1004
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the request-trace-uploader scaffold, closed-segment reader, and existing tests for segment selection and record parsing. Done means v1.4.0 records and failure paths are covered, the S3-compatible backend submits and confirms through the declared interface, and a source segment is deleted only after successful upload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100