huggingface / huggingface/candle

Add batch-native multi-prompt decode loop for heterogeneous LoRA serving

Open
#3,766 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Context

This request builds on #3759 and the open PR chain:

- #3762 adds heterogeneous multi-LoRA batching APIs such as `LoraLinear::forward_with_adapters` and `Llama::forward_with_adapters`.
- #3763 builds on #3762 and adds the CUDA BGMV fast path for decode-step heterogeneous LoRA batching.

Those PRs provide the model/projection side needed for S-LoRA / Punica-style execution, but serving integrations still need a full batch-native autoregressive decode loop that can drive multiple active prompt rows through prefill/decode without falling back to host-side per-adapter sub-batches.

## Request

Please expose or document a batch-native multi-prompt generation/decode path for Llama-family models where each active batch row can select a different LoRA adapter, or no adapter.

The important target is the full generation loop, not only one isolated model forward.

## Required behavior

- Batch prefill accepts multiple prompt rows for the same base model.
- Batch decode accepts one token per active sequence per step.
- Each row can select `Some(adapter_id)` or `None`.
- Logits are returned as `[batch, vocab]`, one row per active sequence.
- KV cache updates remain isolated per sequence.
- Rows with `None` remain strict base-model execution.
- Output matches sequential per-row/per-adapter generation within expected numeric tolerance.
- The path composes with the APIs from #3762 and the CUDA fast path from #3763.

## Acceptance criteria

- A test runs at least three active rows in one decode step: adapter A, adapter B, and no adapter.
- Batched decode output matches sequential reference output.
- The no-adapter row is equivalent to the plain base path.
- The decode loop does not split the batch by adapter on the host side.
- Existing `forward`, `forward_with_adapters`, and sequential generation behavior remain backward-compatible.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the heterogeneous batching APIs from #3762, especially LoraLinear::forward_with_adapters and Llama::forward_with_adapters, then study the CUDA BGMV path from #3763 alongside the existing Llama generation loop. Done means one batch handles adapter A, adapter B, and no adapter through prefill and decode, with isolated KV state and logits matching sequential reference output without host-side adapter splitting.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.