Spike on tracking quality metrics for nvbench comparison scripts
Open
@fbusato is already working on this.
Since Sep 8, 2026.
- Dominant language
- Cuda
- Stars
- 927
- Forks
- 123
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 2
Description
We should investigate how to track quality metrics for perf comparison:
- number of ambiguous comparison results
- number of false positives
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with:
- a list of metrics to be tracked with their descriptions (i.e. number of ambiguities, false positives etc.)
- methodology for getting these metrics (i.e. do I run a script that stores number of ambg? how do I tell a result is false positive?)
We can defer the following to a different issue:
- how to compare two sets of metrics (i.e. number of amb is lower but false positives is larger, is it better?)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.