Lightning-AI / Lightning-AI/litgpt
Initial and final evaluation in `finetune` scripts do not accumulate over devices
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Bug description
I am looking at litgpt/finetune/lora.py and litgpt/finetune/full.py. In the LoRA code, the periodic evaluation L401-418 runs validate on each device, then accumulates parts by all_reduce.
But this does not happen for initial evaluation L310 and final evaluation L260. This seems pretty wrong to me, the loss values would just be the one on the rank 0 device.
Another issue is the val_loss value which is printed in L395, but which seems never updated in L401-418.
I'd be happy to submit a PR fixing all this, but first wanted to check whether I understand something wrong here?
Reproduced in studio
No response
What operating system are you using?
macOS
LitGPT Version
0.5.9
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read litgpt/finetune/lora.py around lines 310 and 401-418, then litgpt/finetune/full.py around line 260, comparing initial and final evaluation with periodic validation. Verify how validation losses are reduced across devices and whether val_loss is refreshed before printing. Done means initial and final values are accumulated consistently and the reported val_loss reflects the corresponding evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100