huggingface / huggingface/candle
Allow loading LoRA adapters into an existing Llama without reloading base weights
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Context
This follows #3759 and #3766.
Tachyon Mesh can now use Candle's heterogeneous LoRA forward/decode APIs from the fork tag `tachyon-v0.11.0-12`, but one integration gap remains for production serving: `LlamaLoadConfig` injects LoRA adapters only while constructing a new `Llama` with `Llama::load_with_config`.
For a long-running server, the base Llama model is already resident in GPU memory. When a batch arrives with tenant-specific LoRA adapters that were not known at base-model load time, the only available public shape is to construct a fresh `Llama` from the base safetensors with those adapters attached. On CUDA this creates a temporary second copy of all base weights plus the adapter weights, which can OOM on models sized near available VRAM. Even when it fits, each batch pays full model-load latency.
Tachyon currently has to disable the CUDA batch-native LoRA path and fall back rather than risk this second GPU allocation.
## Request
Please expose an additive API that allows registering, replacing, and optionally removing LoRA adapters on an already-loaded Llama-family model, without reloading or duplicating the base weights.
A concrete API can differ, but something equivalent to this would work:
```rust
impl Llama {
pub fn load_lora_adapter_prefixed(
&mut self,
name: &str,
vb: VarBuilder,
config: LoraConfig,
prefix: &str,
) -> Result<()>;
pub fn remove_lora_adapter(&mut self, name: &str) -> Result<()>;
}
```
## Required Behavior
- The base model weights remain resident and are not reloaded from safetensors.
- New adapter tensors can be loaded onto the model device and attached to all targeted projections.
- `forward_with_adapters` can use adapters registered after `Llama` construction.
- `set_active_adapter` can select adapters registered after `Llama` construction.
- Replacing an adapter with the same name is either rejected or well-defined.
- Removing an adapter clears it from targeted projections and fails if the adapter is currently merged into base weights.
- Rows with `None` remain strict base-model execution.
- Existing `LlamaLoadConfig` construction-time loading remains backward-compatible.
## Acceptance Criteria
- A test loads a base Llama model, then registers two LoRA adapters after construction.
- A batched forward/decode step runs rows for adapter A, adapter B, and no adapter without constructing a second Llama.
- Outputs match the equivalent construction-time adapter loading path within expected numeric tolerance.
- A CUDA test or benchmark demonstrates that dynamic adapter registration does not allocate a second copy of the base weights.
- The API composes with the heterogeneous batching work in #3759 and the batch-native decode loop requested in #3766.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with LlamaLoadConfig and Llama::load_with_config, then trace how adapters are attached and used by forward_with_adapters and set_active_adapter. Compare the construction-time path with the heterogeneous batching work in #3759 and the batch-native decode loop in #3766. Done means post-construction registration, selection, replacement, and removal work without reloading base weights, with tests covering adapter A, adapter B, and no adapter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100