Lightning-AI / Lightning-AI/litgpt
TPU Pod Training
Open
Nobody has claimed this yet.
help wanted
question
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Hello,
I'm trying to pre-train a llama model using fabric on a TPU Pod. I have access to a few v4-32s. training on a v4-8 is trivial using pytorch XLA on TPUs but scaling to a pod is giving me issues.
LitGPT seems like the most promising pytorch XLA based framework to go with. Can you guys help me with this?
Thanks,
Omead
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue mentions LitGPT, Fabric, PyTorch XLA, and TPU Pods but names no files, tests, or entry points. First clarify the expected TPU Pod training behavior and identify the relevant training entry point; completion is not defined in the current request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100