Lightning-AI / Lightning-AI/pytorch-lightning
Improve Fault Tolerance via TorchFT
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
https://github.com/pytorch/torchft
https://pytorch.org/blog/fault-tolerant-llama-training-with-2000-synthetic-failures-every-15-seconds-and-no-checkpoints-on-crusoe-l40s/
I would like there to be able to use torchft to improve my fault tolerance inside of pytorch lightning.
### Pitch
_No response_
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked TorchFT repository and PyTorch blog post, then review how PyTorch Lightning currently handles fault tolerance. Define the integration scope and a concrete validation scenario before identifying implementation files or tests; the issue is done only when TorchFT-based fault tolerance is usable inside PyTorch Lightning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100