explosion / explosion/curated-transformers

Add Low-Rank Adapters injection into base models

Open
#312 3 comments 1 reaction 0 assignees View on GitHub
feat/training type/feature
Dominant language
Python
Stars
890
Forks
36
PR merge metrics
No merged PRs in 30d

Description

Low-Rank Adaptation (LoRA) has become the de-facto parameter-efficient finetuning technique to adapt a base language model to a specific task. `curated-transformers` already supports dynamic quantization using `bitsandbytes`, hence adding some utilities to inject trainable adapters opens the door to using `curated-transformers` as a replacement to the HuggingFace `transformers` + `peft` stack. This could also enable a very nice finetuning integration into spaCy in the future.

For reference, I find [this implementation](https://github.com/Lightning-AI/lit-gpt/blob/main/lit_gpt/lora.py) in [`lit-gpt`](https://github.com/Lightning-AI/lit-gpt) really readable.

Do you find this idea interesting?

If so, as for the user-facing API, drawing inspiration from HuggingFace `peft` it could look something like

```python
# Load and quantize the base model
model = AutoGenerator.from_hf_hub(
name="meta-llama/Llama-2-7b-chat-hf",
device=torch.device("cuda", index=0),
quantization_config=BitsAndBytesConfig.for_4bit(
quantization_dtype=Dtype4Bit.FP4,
compute_dtype=torch.bfloat16,
double_quantization=True,
),
)

# Replace targeted linear layers by `LoRALayer` that wrap the original weights
model_with_adapters = inject_adapters(
base_model=model,
lora_config=LoraConfig(
rank=64,
alpha=16,
dropout=0.1,
bias=LoraBias.NONE,
target_modules=[...]
),
)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.