[FEA] Add reproducible benchmark harness foundation
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 36
- Forks
- 11
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 25
Description
Is your feature request related to a problem? Please describe.
DAQIRI benchmark runners do not share a reproducible lifecycle or result contract. A zero process exit can be mistaken for a successful measurement even when required application, kernel, NIC, or physical-path evidence is absent, and cross-host cleanup does not identify exact remote processes.
Describe the solution you'd like
Add a consolidated benchmark-harness foundation with one stable Python entrypoint, versioned experiment and site-profile schemas, deterministic plan/dry-run output, immutable resolved provenance, run-ID-based local/SSH process ownership, canonical valid/invalid/failed results, and derived CSV/Markdown reports. The first supported adapter should be externally configured physical cross-host UDP, with active-window counters, explicit placement/rate scope, strict loss-free validation, documentation, and hardware-free lifecycle tests.
Describe alternatives you've considered
Extending the existing shell controllers would retain fragmented schemas and transport-specific lifecycle behavior. Migrating every transport at once would make the change too broad to review safely.
Additional context
Follow-up issues can add transactional topology management, loopback modes, raw Ethernet, TCP/RDMA/workloads, and ResNet adapters. This request does not change published benchmark values and is separate from #294/#295 and the GPU-workload performance investigation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping the existing shell controllers and benchmark lifecycle or schema entry points, then define the scope of the stable Python entrypoint and first externally configured physical cross-host UDP adapter. Review the requested hardware-free lifecycle tests and deterministic dry-run output. Done means the foundation covers versioned schemas, immutable provenance, run-ID ownership, canonical results, validation, and derived reports without changing published benchmark values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking, performance, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100