mlcommons / mlcommons/inference
Must preview results be followed by a published result or is a submission + withdrawal acceptable?
Open
Nobody has claimed this yet.
postmortem 5.0
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
- There have been cases where competitive results got changed and even though available submissions were ready, the company was not happy publishing them.
- One potential solution here will be for the preview system submitter to show the performance to the review committee and make the code and hardware available in the subsequent round. Whether the preview system must now be audited is a question to be answered.
Post mortem task proposed by @arjunsuresh
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the post mortem task proposed by @arjunsuresh and compare the two scenarios described: publishing a preview result versus submitting and withdrawing it. Done means the review committee’s required process is decided and the outcome is recorded clearly, including whether preview systems must be audited and when code and hardware become available.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100