Lightning-AI / Lightning-AI/pytorch-lightning
Avoid network access during ordinary tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## Proposed refactor
Some of our tests use the `MNIST` and `TrialMNIST` classes which download the MNIST data off the internet (or from a cache).
https://github.com/PyTorchLightning/pytorch-lightning/blob/948cfd24de4f64a2980395581f15544e5e37eab0/tests/helpers/datasets.py#L25
https://github.com/PyTorchLightning/pytorch-lightning/blob/948cfd24de4f64a2980395581f15544e5e37eab0/tests/helpers/datasets.py#L134
### Motivation
We should avoid this to reduce the inherent flakiness of network access.
### Pitch
If a test uses these classes, do one of 3 options:
- Remove the test, if we consider it's not useful (probably an old test)
- Update the test to use either `RandomDataset` or a `RandomMNIST` class which would "mock" the actual MNIST dataset but with random data.
- Move the test to `tests/benchmarks`
### Additional context
Master is currently blocked due to errors while reading the MNIST data zipfiles.
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @borda @akihironitta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tests/helpers/datasets.py at the MNIST and TrialMNIST definitions, then find ordinary tests that use them. Review each usage and decide whether it should be removed, use RandomDataset or a random MNIST substitute, or move to tests/benchmarks. Done means ordinary tests no longer access the network or depend on MNIST downloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100