mlcommons / mlcommons/inference
[4.1 postmortem] Unit test module for inference repo
Open
@pgmpablo157321 is already working on this.
Since Oct 8, 2024.
postmortem 4.1
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
We propose to add some basic unit test framework (likely pytest) and tests to the inference repo. Ideally, it should test:
- All configuration (mlperf.conf, user.conf) is valid and working (i.e. without typo, no items outside of an existing dictionary etc)
- All reference benchmark dataset and model download modules are working
- All reference models are running and producing reasonable output
- Submission checker can catch errors as expected.
- etc..
To get this started, we can aim at some simple tests:
- configuration walkthrough and validation
- submission checker tests for some corner cases (once #1670 has started)
@pgmpablo157321 to help implement
@nv-alicheng for more suggestions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.