Collapse reference+learner hydra heads when using LoRa
Open
contributions welcome
feature request
good first issue
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### 🚀 The feature, motivation, and pitch
With additive (delta-style) parameter-efficient tuning methods such as [LoRa](https://arxiv.org/abs/2106.09685), we should be able to make a slightly more mem-efficient hydra architecture by using a single block that does ~`frozen_head + tunable_weights` for the learner/policy head's fwd-pass and simply `frozen_head` for the reference, instead of maintaining 2x heads.
CC @LouisCastricato and @cat-state for pointing this out
### Alternatives
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.