huggingface / huggingface/candle

Add first-class LoRA adapter injection to candle-transformers model loaders

Open
#3,696 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Motivation

`candle-nn` now has a useful `LoraLinear` primitive for PEFT-style adapters, but `candle-transformers` model implementations such as Llama cannot currently consume it without downstream projects reimplementing the model forward pass.

In `candle_transformers::models::llama::Llama`, the transformer blocks and projection fields are private, so a downstream runtime cannot replace `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, or `down_proj` with `LoraLinear` wrappers after loading the base model.

This forces consumers to duplicate Candle's Llama forward implementation just to apply LoRA adapters, which is fragile and makes it harder to stay aligned with upstream fixes.

## Use case

Tachyon-Mesh serves one shared foundation model for multiple tenants. Each request can include an `adapter_id`, resolved to a PEFT `.safetensors` LoRA adapter stored by the model broker.

Example flow:

1. Load one base Llama safetensors checkpoint once.
2. Tenant A sends a request with `adapter_id = "legal-v1"`.
3. Tenant B sends a request with `adapter_id = "support-v2"`.
4. The runtime hot-swaps the active LoRA adapter per request without reloading the base model.
5. Requests without `adapter_id` run against the unmodified base model.

Today, implementing this requires a local Llama forward copy because `candle-transformers::Llama` does not expose an adapter injection point.

## Proposed API direction

Please add a first-class way for `candle-transformers` loaders to build supported models with optional LoRA adapters.

One possible shape:

```rust
use candle_nn::lora::LoraConfig;
use candle_transformers::models::llama::{Llama, LlamaLoadConfig};

let base_vb = VarBuilder::from_mmaped_safetensors(&base_paths, DType::F32, &device)?;
let adapter_vb = VarBuilder::from_mmaped_safetensors(&adapter_paths, DType::F32, &device)?;

let load_config = LlamaLoadConfig::default().with_lora_adapter(
"legal-v1",
adapter_vb,
LoraConfig {
rank: 16,
alpha: 32.0,
target_modules: vec![
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
},
);

let mut model = Llama::load_with_config(base_vb, &config, load_config)?;
model.set_active_adapter(Some("legal-v1"))?;

let logits = model.forward(&input, index_pos, &mut cache)?;
```

The exact API can differ, but the important requirements are:

- Load PEFT `.safetensors` tensors named `lora_A.weight` and `lora_B.weight`.
- Apply adapters to attention and MLP projections.
- Allow multiple named adapters to be registered.
- Allow per-request active adapter selection.
- Allow disabling adapters for base-model inference.
- Avoid requiring downstream users to copy private model internals.
- Preserve the existing `Llama::load` behavior for users who do not need LoRA.

## Alternative API

Instead of a Llama-specific config, Candle could expose a more generic projection factory/hook used by model loaders:

```rust
trait LinearFactory {
fn linear_no_bias(
&self,
in_dim: usize,
out_dim: usize,
path: &str,
vb: VarBuilder,
) -> Result>;
}
```

Then model loaders could use this factory for projections, and downstream runtimes could wrap selected projections with `LoraLinear`.

## Why this belongs upstream

LoRA support is a common serving requirement, and the model graph knows exactly where the projection modules are. Keeping this in `candle-transformers` would:

- prevent duplicated model forward implementations downstream;
- keep downstream runtimes aligned with Candle's cache, RoPE, attention, and architecture fixes;
- make `candle-nn::LoraLinear` directly useful for real transformer inference;
- enable multi-adapter serving and hot-swap use cases in a standard way.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with candle_nn::lora::LoraLinear and the candle_transformers::models::llama::Llama loader and projection fields. Trace how Llama constructs attention and MLP projections, then determine an API for loading named PEFT safetensors adapters and selecting or disabling them per request. Done means existing Llama::load behavior is preserved while multiple adapters can target the listed projections without downstream forward-pass copies.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.