NVIDIA / NVIDIA/daqiri

[FEA] Add reproducible benchmark harness foundation

Open
#296 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
36
Forks
11
Avg merge
3d 4h
Merged PRs (30d)
25

Description

Is your feature request related to a problem? Please describe.

DAQIRI benchmark runners do not share a reproducible lifecycle or result contract. A zero process exit can be mistaken for a successful measurement even when required application, kernel, NIC, or physical-path evidence is absent, and cross-host cleanup does not identify exact remote processes.

Describe the solution you'd like

Add a consolidated benchmark-harness foundation with one stable Python entrypoint, versioned experiment and site-profile schemas, deterministic plan/dry-run output, immutable resolved provenance, run-ID-based local/SSH process ownership, canonical valid/invalid/failed results, and derived CSV/Markdown reports. The first supported adapter should be externally configured physical cross-host UDP, with active-window counters, explicit placement/rate scope, strict loss-free validation, documentation, and hardware-free lifecycle tests.

Describe alternatives you've considered

Extending the existing shell controllers would retain fragmented schemas and transport-specific lifecycle behavior. Migrating every transport at once would make the change too broad to review safely.

Additional context

Follow-up issues can add transactional topology management, loopback modes, raw Ethernet, TCP/RDMA/workloads, and ResNet adapters. This request does not change published benchmark values and is separate from #294/#295 and the GPU-workload performance investigation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by mapping the existing shell controllers and benchmark lifecycle or schema entry points, then define the scope of the stable Python entrypoint and first externally configured physical cross-host UDP adapter. Review the requested hardware-free lifecycle tests and deterministic dry-run output. Done means the foundation covers versioned schemas, immutable provenance, run-ID ownership, canonical results, validation, and derived reports without changing published benchmark values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking, performance, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.