huggingface / huggingface/candle
`candle-transformers::generation`: provide stop criteria and a finish reason
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
## Problem
`generation` covers producing a token — `Sampling`, `LogitsProcessor`,
`apply_frequency_presence_penalty`, and `IncrementalDecoder` (#3789). It has
nothing for the other half of a decode loop: deciding when to stop, and saying
why it stopped.
So every serving loop writes that part itself, and it is harder than it looks.
A stop sequence is a property of the *text*, not of the token ids, so it has to
be scanned over incrementally decoded output; it can straddle a token boundary,
so a suffix must be withheld until it is known not to be the start of one; what
is emitted must land on a char boundary; and with several stop sequences the
earliest match has to win, not the first one checked.
The finish reason is the part that is quietly wrong most often. Inferring it
from the token count alone — `completion_tokens >= max_new_tokens` means
`length` — is wrong exactly at the boundary, because a generation whose EOS or
stop sequence lands on the budget's last token has both spent its budget and
ended normally. Reporting `length` there tells an OpenAI-compatible client that
a complete answer was truncated, which is the same misreport as the opposite
error, in the other direction.
This is not hypothetical. In Tachyon-Mesh four decode loops — ordinary,
speculative, continuous-batch, and a batch-native LoRA loop — each grew their
own copy of this logic, and a review pass found the same class of defect in
every one of them: the boundary case above in all four, and in two of them a
withheld separator that was either dropped from the stream or emitted twice,
because "bytes accounted for" and "bytes actually sent" had been conflated into
one counter. Each loop got it wrong differently, which is the signature of
logic that should exist once.
## What a correct implementation needs
1. **Scan text, not ids.** The needle is a string; tokenization is irrelevant to
whether it matched.
2. **Withhold a suffix** at least as long as the longest stop sequence, so a
match split across two tokens is still found.
3. **Emit only to a char boundary**, since the held-back window is measured in
bytes and multi-byte characters straddle it.
4. **Earliest match wins** across all configured sequences — checking in
configuration order truncates at the wrong place.
5. **Distinguish "the model ended it" from "the budget ended it"**, and let the
former win when both are true on the same token.
6. **Report nothing rather than guess.** A loop that stopped for a reason it
cannot name (a wall-clock deadline, a departed consumer) should leave the
reason absent instead of picking the closest label.
## Proposed API
```rust
pub struct StopCriteria { /* … */ }
impl StopCriteria {
pub fn new(stop: impl IntoIterator, eos: Vec) -> Self;
/// Bytes to withhold: the longest stop sequence, minus one.
pub fn hold(&self) -> usize;
/// Byte offset of the earliest match in `text`, if any.
pub fn matched(&self, text: &str) -> Option;
/// Largest char-boundary offset safe to emit from `text` right now.
pub fn safe_emit_end(&self, text: &str) -> usize;
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum FinishReason {
/// EOS, or a configured stop sequence.
Stop,
/// The token budget ran out first.
Length,
}
```
Used with `IncrementalDecoder`, a loop becomes: push the token, ask
`matched()`, emit up to `safe_emit_end()`, and report `Stop` whenever the model
ended the sequence itself — with `Length` reserved for the case where it did
not.
## Note
This pairs with #3789: `IncrementalDecoder` gives the text, and this decides
what to do with it. The two were built together and it may be worth reviewing
them that way.
I am happy to submit the PR, including the boundary cases above as tests —
particularly EOS landing on the budget's last token, and a stop sequence split
across two tokens.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the candle-transformers::generation entry point and review how IncrementalDecoder supplies text. Add the proposed StopCriteria and FinishReason API, covering incremental stop-sequence matching, character-safe emission, earliest-match selection, and EOS-versus-budget precedence. Validate the boundary cases described in the issue, especially a stop sequence split across tokens and EOS on the final budget token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100