CarperAI / CarperAI/trlx

Migrate to `peft` from `opendelta` for parameter efficient tuning methods

Open
#434 6 comments 2 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

Let's migrate to [`peft`](https://github.com/huggingface/peft).

##### Tasks
Doing so will require the following updates:

1. Replace the `opendelta` setup in the `AccelerateBaseTrainer` with a `peft` backed setup: https://github.com/CarperAI/trlx/blob/92b68e4d8c5d59e6ba25d12fd9acfe10287be689/trlx/trainer/accelerate_base_trainer.py#L145-L155

2. Handle fine-grained layer capturing to only modify the upper trunk layers of hydra architectures as handled below:
https://github.com/CarperAI/trlx/blob/92b68e4d8c5d59e6ba25d12fd9acfe10287be689/trlx/utils/modeling.py#L414-L428

##### Motivation
Citing @ethankim00's concerns with `opendelta`:
- `opendelta` import fails due to an unnecessary `turtle` package import. Even if pip installed, users may be required to have `sudo` privileges to install the corresponding base graphics package https://github.com/thunlp/OpenDelta/issues/47
- Doesn’t seem to work with DeepSpeed ZeRO 3
- Additional inference overhead from not merging in the LoRA adapters layers
- Incompatibility with int8 training
- Less actively maintained than the `peft` library, which has been growing rapidly
- Sharing adapter weights on the HuggingFace Hub is less convenient with `opendelta`

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.