Lightning-AI / Lightning-AI/lit-llama
multi gpus for full finetune
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 517
- PR merge metrics
- No merged PRs in 30d
Description
I want to run the full.py on multi gpus, but only one GPU was used.
```
Using bfloat16 Automatic Mixed Precision (AMP)
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------
[rank: 0] Global seed set to 1337
```
Can you help me to solve this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with full.py and the distributed initialization output shown in the issue, then check how the script is launched and how many processes are configured. Reproduce the run with multiple GPUs; done when initialization reports one process per GPU and full fine-tuning uses all requested devices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100