AnswerDotAI / AnswerDotAI/fsdp_qlora
DoRA training not taking dropout or alpha into account
- Dominant language
- Jupyter Notebook
- Stars
- 1.6k
- Forks
- 201
- PR merge metrics
- No merged PRs in 30d
Description
I think there is a bug in the DoRA implementation as it takes neither `lora_dropout` nor `lora_alpha` into account. These arguments are passed as `*args` to the `__init__` call of the DoRA layers but subsequently ignored inside of [dora.py](https://github.com/AnswerDotAI/fsdp_qlora/blob/05ed9f2a60f96a0795cb082bceab70a9b19fd213/scripts/dora.py). This can be easily missed as the DoRA paper does not include them in their equations, but they are mentioned elsewhere in the paper and should be applied the same as in the LoRA implementation.
Also note that `lora_dropout` is only applied to the LoRA/DoRA output, not the [base model output](https://github.com/AnswerDotAI/fsdp_qlora/blob/05ed9f2a60f96a0795cb082bceab70a9b19fd213/scripts/dora.py#L92), which I believe has an impact on [these lines](https://github.com/AnswerDotAI/fsdp_qlora/blob/05ed9f2a60f96a0795cb082bceab70a9b19fd213/scripts/dora.py#L111-L113), as they currently assume that the same `x` is used for the base layer and the DoRA part.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in scripts/dora.py, especially the DoRA layer __init__ and the code around lines 92 and 111-113. Compare how the LoRA implementation handles lora_dropout and lora_alpha, then trace how the base-model and DoRA outputs use x. Done means both arguments affect DoRA behavior and the output paths no longer rely on an incorrect shared input assumption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100