huggingface / huggingface/candle
[feature] First-class LoRA adapters in candle-nn (load + merge + multi-adapter)
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
### Motivation
LoRA is ubiquitous for serving fine-tuned variants, but candle has no in-tree
support, so every consumer re-implements `W·x + (B·A)·x·(α/r)` and PEFT loading.
A reusable `candle-nn` primitive would standardize this and enable multi-adapter
serving / hot-swap. (Out-of-tree prior art: `EricLBuehler/candle-lora`.)
### Gap in candle
- No `lora` module in `candle-nn`.
### Proposed implementation
- `candle-nn::LoraLinear` wrapping a base `Linear` with `A [r, in]`, `B [out, r]`
and `alpha/r` scaling.
- Load PEFT `.safetensors` adapters via `VarBuilder`.
- `merge()` / `unmerge()` (fold weights for inference) and active-adapter
selection for multi-LoRA.
### Use-case / example
```rust
// Apply a PEFT adapter on top of a base projection at load time.
let base = linear(in_dim, out_dim, vb.pp("q_proj"))?;
let lora = LoraLinear::from_peft(base, vb_adapter.pp("q_proj"), /*rank=*/16, /*alpha=*/32.0)?;
let y = lora.forward(&x)?; // base + LoRA delta
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading candle-nn's existing Linear and VarBuilder APIs, then compare them with the proposed LoraLinear shape and PEFT safetensors loading. Define how the base projection, A/B matrices, alpha/r scaling, and active adapters interact. Done means the issue's forward, load, merge/unmerge, and multi-adapter behaviors are supported with a clear usage example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100