huggingface / huggingface/candle
Add batch-native multi-prompt decode loop for heterogeneous LoRA serving
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Context
This request builds on #3759 and the open PR chain:
- #3762 adds heterogeneous multi-LoRA batching APIs such as `LoraLinear::forward_with_adapters` and `Llama::forward_with_adapters`.
- #3763 builds on #3762 and adds the CUDA BGMV fast path for decode-step heterogeneous LoRA batching.
Those PRs provide the model/projection side needed for S-LoRA / Punica-style execution, but serving integrations still need a full batch-native autoregressive decode loop that can drive multiple active prompt rows through prefill/decode without falling back to host-side per-adapter sub-batches.
## Request
Please expose or document a batch-native multi-prompt generation/decode path for Llama-family models where each active batch row can select a different LoRA adapter, or no adapter.
The important target is the full generation loop, not only one isolated model forward.
## Required behavior
- Batch prefill accepts multiple prompt rows for the same base model.
- Batch decode accepts one token per active sequence per step.
- Each row can select `Some(adapter_id)` or `None`.
- Logits are returned as `[batch, vocab]`, one row per active sequence.
- KV cache updates remain isolated per sequence.
- Rows with `None` remain strict base-model execution.
- Output matches sequential per-row/per-adapter generation within expected numeric tolerance.
- The path composes with the APIs from #3762 and the CUDA fast path from #3763.
## Acceptance criteria
- A test runs at least three active rows in one decode step: adapter A, adapter B, and no adapter.
- Batched decode output matches sequential reference output.
- The no-adapter row is equivalent to the plain base path.
- The decode loop does not split the batch by adapter on the host side.
- Existing `forward`, `forward_with_adapters`, and sequential generation behavior remain backward-compatible.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the heterogeneous batching APIs from #3762, especially LoraLinear::forward_with_adapters and Llama::forward_with_adapters, then study the CUDA BGMV path from #3763 alongside the existing Llama generation loop. Done means one batch handles adapter A, adapter B, and no adapter through prefill and decode, with isolated KV state and logits matching sequential reference output without host-side adapter splitting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100