huggingface / huggingface/candle

[feature] Add min_p sampling and frequency/presence penalties to generation

Open
#3,654 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

### Motivation
`candle_transformers::generation::Sampling` offers ArgMax, All, TopK, TopP,
TopKThenTopP and GumbelSoftmax, but not **min_p** (a popular, robust alternative
to top_p) nor OpenAI-style **frequency_penalty / presence_penalty**. These are
expected by anyone exposing an OpenAI-compatible API on top of candle.

### Gap in candle
- No `min_p` variant in `Sampling`.
- No frequency/presence penalty helpers in the generation module (repeat-penalty,
where present, lives outside this module and differs from OpenAI semantics).

### Proposed implementation
- Add `Sampling::MinP { p: f64, temperature: f64 }` (keep min-p tokens with prob
≥ `p * max_prob`), wired through `LogitsProcessor`.
- Add penalty helpers applied to logits before sampling, parameterized over the
generated-token history, matching OpenAI's frequency/presence semantics.

### Use-case / example
```rust
// min_p sampling for an OpenAI-compatible server.
let mut lp = LogitsProcessor::from_sampling(seed, Sampling::MinP { p: 0.05, temperature: 0.8 });
let next = lp.sample(&logits)?;

// frequency / presence penalties over the running output.
let logits = apply_frequency_presence_penalty(&logits, &generated, freq: 0.5, presence: 0.3)?;
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with candle_transformers::generation::Sampling and LogitsProcessor, tracing how existing sampling variants are wired. Check the generation module's tests or sampling entry points before implementing MinP and the frequency/presence penalty helpers. Done means both requested behaviors are exposed, match the stated semantics, and are covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.