NVIDIA / NVIDIA/nvcf

feat(request-trace-uploader): add configurable per-record-type sampling

Open
#1,444 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Why

High-volume deployments cannot export every record. Sampling must be coherent: a request's payload record and its metadata record have to be kept or dropped together, or the export is useless for both audit and replay.

Scope

  • Apply sampling after record suppression. Suppressed and non-suppressed records are sampled identically.
  • Declare the sampling key per record type rather than as a cross-type fallback chain. request_end uses request.request_id, request_payload uses payload.request_id, and the three tool events use agent_context.session_id. A fallback chain across these would mix identifier namespaces and break coherence, because a session contains many requests.
  • Make the decision a deterministic function of the key alone, so records sharing a key are kept or dropped identically across record types, segments, and uploader restarts. Per-record randomness is not acceptable: a request's payload and its request_end are written to the same file family but may land in different segments uploaded at different times, and determinism is what makes them agree without shared state.
  • Document that tool events carry no request identifier, so tool sampling is coherent at session granularity and does not join to request-level sampling.
  • Define policy for records with no resolvable key: keep, drop, or reject at config validation. Default keep.
  • Emit records_sampled_out_total by record type and reason, where reason is rate or no_key.

Tests

  • Deterministic keep or drop for the same key across record types, segments, and process restarts.
  • A request's request_payload and request_end receive the same decision.
  • Tool events key on session and are unaffected by request-level decisions.
  • Missing-key policy under each setting.
  • Rate boundaries at 0.0 and 1.0.

Note

Record-level sampling requires rewriting each segment, which forfeits byte-exact jsonl.gz passthrough on the object-store backend. If passthrough matters more than granularity, segment-level sampling is the alternative, but it cannot give coherent per-request retention.

Parent: #1004

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or existing tests; start by locating the request-trace uploader and definitions for request_end, request_payload, and the tool events. Read parent issue #1004, then identify where suppression, segment rewriting, configuration validation, and object-store passthrough are handled. Done means deterministic per-key sampling, missing-key policy, metrics, documentation, and the listed boundary and cross-record tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.