CarperAI / CarperAI/trlx

Collapse reference+learner hydra heads when using LoRa

Open
#320 6 comments 2 reactions 0 assignees View on GitHub
contributions welcome feature request good first issue
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

With additive (delta-style) parameter-efficient tuning methods such as [LoRa](https://arxiv.org/abs/2106.09685), we should be able to make a slightly more mem-efficient hydra architecture by using a single block that does ~`frozen_head + tunable_weights` for the learner/policy head's fwd-pass and simply `frozen_head` for the reference, instead of maintaining 2x heads.

CC @LouisCastricato and @cat-state for pointing this out

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.