explosion / explosion/curated-transformers
Optimal Qlora settings
Open
feat/training
type/feature
- Dominant language
- Python
- Stars
- 890
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
In HF transformers, the default setting of qlora does not replicate the qlora of the original paper, leaving valuable performance lying on the ML practitioners street using lib defaults.
One has to apply lora to certain parts of the NN, please see Tweet by Tim Dettmers:
https://twitter.com/Tim_Dettmers/status/1695377756232589459
I guess this has to be customized for each model architecture, sounds like a feature for curated-transformers, to me.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.