Lightning-AI / Lightning-AI/pytorch-lightning
Decouple benchmarks from tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
I've removed the convergence check. It was failing under batch mode. There might be RNG interactions between tests
results = trainer.test(datamodule=dm)
> assert results[0]["test_acc"] > 0.7
E assert 0.6937500238418579 > 0.7
It's really arbitrary and PyTorch doesn't even guarantee performance between versions
These kinds of tests should be benchmark tests. And ideally something like the same model trained with DeepSpeed vs normally
Originally posted by @carmocca in https://github.com/Lightning-AI/lightning/pull/13673#discussion_r922250654
This will contribute to less flakiness in our CI 🚀
cc @borda @akihironitta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the test containing trainer.test(datamodule=dm) and the convergence check described in the issue. Review the existing benchmark-test structure and how these tests run in CI. Done means performance assertions no longer make regular tests flaky, with benchmark coverage kept separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100