huggingface / huggingface/candle

[feature] First-class LoRA adapters in candle-nn (load + merge + multi-adapter)

Open
#3,652 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

### Motivation
LoRA is ubiquitous for serving fine-tuned variants, but candle has no in-tree
support, so every consumer re-implements `W·x + (B·A)·x·(α/r)` and PEFT loading.
A reusable `candle-nn` primitive would standardize this and enable multi-adapter
serving / hot-swap. (Out-of-tree prior art: `EricLBuehler/candle-lora`.)

### Gap in candle
- No `lora` module in `candle-nn`.

### Proposed implementation
- `candle-nn::LoraLinear` wrapping a base `Linear` with `A [r, in]`, `B [out, r]`
and `alpha/r` scaling.
- Load PEFT `.safetensors` adapters via `VarBuilder`.
- `merge()` / `unmerge()` (fold weights for inference) and active-adapter
selection for multi-LoRA.

### Use-case / example
```rust
// Apply a PEFT adapter on top of a base projection at load time.
let base = linear(in_dim, out_dim, vb.pp("q_proj"))?;
let lora = LoraLinear::from_peft(base, vb_adapter.pp("q_proj"), /*rank=*/16, /*alpha=*/32.0)?;
let y = lora.forward(&x)?; // base + LoRA delta
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading candle-nn's existing Linear and VarBuilder APIs, then compare them with the proposed LoraLinear shape and PEFT safetensors loading. Define how the base projection, A/B matrices, alpha/r scaling, and active adapters interact. Done means the issue's forward, load, merge/unmerge, and multi-adapter behaviors are supported with a clear usage example.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.