huggingface / huggingface/alignment-handbook
Training on LORA using multi-gpu is giving constant loss
Open
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 490
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
I am trying to train yi-34B model using LORA setup on multi-gpu. But i am getting constant loss i.e. around 2 throughout my SFT training on 4 epochs. And inferencing trained model is giving useless response.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.