huggingface / huggingface/alignment-handbook
Does QLora DPO Training support reference model?
Open
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 490
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Hello! Thanks for your awesome work!
I meet an issue when I run dpo with qlora. I notice there is a setting:
```
if model_args.use_peft is True:
ref_model = None
ref_model_kwargs = None
```
I also notice that the `use_peft` is set to true only in config_qlora.yaml. This means if we use qlora to do dpo training, we do not use reference model at all.
I wonder if this code support qlora training with reference model? Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.