huggingface / huggingface/candle
Add heterogeneous LoRA batching kernels for S-LoRA/Punica-style multi-adapter forward
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Context
Tachyon Mesh issue #316 implements phase 1 of Multi-LoRA serving on the host side: requests for the same base model are now grouped into one scheduler step, then split into adapter-specific sub-batches before calling Candle.
This preserves correctness and avoids splitting scheduler steps by `adapter_id`, but it still executes each adapter group sequentially. To reach the target throughput for 5-10 concurrent agents using distinct LoRA adapters on the same resident base model, Candle needs a true heterogeneous LoRA forward path.
## Request
Please add S-LoRA/Punica-style heterogeneous LoRA batching support for Llama-family safetensors models.
Needed capability:
- One base model forward receives a batch containing multiple `adapter_id`s.
- Each sequence can select a different active LoRA adapter.
- LoRA A/B projections are applied through batched/segmented SGMV-style kernels rather than host-side per-adapter forwards.
- The API preserves the current no-adapter path unchanged.
- The implementation works with the existing Tachyon fork APIs around `LlamaLoadConfig`, `LoraConfig`, `set_active_adapter`, and paged-attention/CUDA graph seams where possible.
## Proposed API shape
A concrete shape can differ, but Tachyon needs something equivalent to:
```rust
model.forward_with_adapters(
input_ids,
index_pos,
cache,
adapter_assignments, // per sequence: None or adapter handle/id
)
```
or a cache/model method that binds a per-row adapter assignment tensor/table before `forward`.
## Acceptance criteria
- A single forward can process a batch with at least two different LoRA adapters and one no-adapter row.
- Outputs match the current sequential per-adapter execution within expected numeric tolerance.
- No-adapter rows are not affected by active adapter state.
- Adapter isolation is covered by a regression test.
- CUDA implementation includes a benchmark or test demonstrating improved aggregate throughput versus sequential sub-batches.
- The API remains additive so Tachyon can keep using the existing sequential fallback until this path is enabled.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing LlamaLoadConfig, LoraConfig, set_active_adapter, and paged-attention/CUDA graph seams, then trace the current sequential per-adapter forward path. Add an additive heterogeneous forward entry point supporting mixed adapters and no-adapter rows. Done means outputs match sequential execution, adapter isolation has a regression test, and CUDA includes a throughput comparison against sequential sub-batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100