Add hparams and compliance checks for training and eval samples for all benchmarks
Open
@shangw-nvidia is already working on this.
Since Nov 10, 2021.
- Dominant language
- Python
- Stars
- 43
- Forks
- 60
- Avg merge
- 19h 56m
- Merged PRs (30d)
- 3
Description
According to issue https://github.com/mlcommons/submission_training_1.0/issues/39, the number of training samples is 117266. Many submissions do hardcode this value, even though the reference does not. On that specific issue the submission in question used a different value.
The decision was to add train_samples, and eval_samples as hyperparameters + the related compliance checker rules so we avoid such issues in the future.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.