NVIDIA / NVIDIA/nvbench

Spike on tracking quality metrics for nvbench comparison scripts

Open
#415 0 comments 0 reactions 1 assignee View on GitHub

@fbusato is already working on this.

Since Sep 8, 2026.

Dominant language
Cuda
Stars
927
Forks
123
Avg merge
2d 10h
Merged PRs (30d)
2

Description

We should investigate how to track quality metrics for perf comparison:

  • number of ambiguous comparison results
  • number of false positives

After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.

This issue can be close with:

  • a list of metrics to be tracked with their descriptions (i.e. number of ambiguities, false positives etc.)
  • methodology for getting these metrics (i.e. do I run a script that stores number of ambg? how do I tell a result is false positive?)

We can defer the following to a different issue:

  • how to compare two sets of metrics (i.e. number of amb is lower but false positives is larger, is it better?)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.