mlcommons / mlcommons/algorithmic-efficiency
Incorrect Imagenet evals with pytorch_eval_num_workers > 0
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 425
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
AlgoPerf submitter team reports that they are no longer able to reproduce the NAdam baseline results in PyTorch using the current repo in PyTorch on the ImageNet workloads (both ResNet and ViT).
See the plot below in terms of differences in the training/validation loss and accuracy between the given NAdam Jax results and the current run's results on ImageNet ViT.
They did not see a change in OGBG and FastMRI.
The list of commits that we merged were from 389fe3f823a5016289b55b48aa8061a37b18b401 to 79ccc5e860d7928cf896ffe12ec686c72fd840d4.
Steps to Reproduce
Running submission runner with eval_num_workers=4 (recently changed default to help speed up evals).
Source or Possible Fix
Setting the eval_num_workers to 0 resolves the discrepancy in evals. We are still investigating why.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the submission runner and the evaluation path controlled by eval_num_workers; reproduce the ImageNet ResNet and ViT discrepancy with eval_num_workers=4, then compare it with 0. Done means the PyTorch NAdam baseline results are reproducible with parallel evaluation without changing the OGBG or FastMRI behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100