cactus-compute / cactus-compute/cactus
Add custom vocabulary / hotword biasing for transcription
- Dominant language
- C++
- Stars
- 6k
- Forks
- 501
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 4
Description
## Summary
Add support for passing a custom vocabulary (hotword list) to the transcription APIs so that domain-specific words, proper nouns, and jargon are transcribed correctly. Today, both Whisper and Moonshine models run with no way to bias the decoder toward expected words, leading to frequent misspellings of names, acronyms, and technical terms.
## Motivation
On-device transcription is a core Cactus feature (Whisper-Small, Whisper-Medium, Moonshine-Base). In real-world apps — medical, legal, enterprise — users need names like "Cactus Compute", drug names like "Omeprazole", or acronyms like "HIPAA" to be recognized accurately. Without hotword biasing the models hallucinate phonetically similar but incorrect words.
Whisper's `prompt` parameter already provides weak conditioning (the user passes a text prompt that seeds the decoder), but it has limits:
- The prompt is tokenized once and prepended; it cannot scale to hundreds of domain terms.
- It offers no per-token probability boost, so rare words are still suppressed by the language model head.
A dedicated custom-vocabulary mechanism would be more reliable and ergonomic.
## Proposed API Changes
### C FFI (`cactus_ffi.h`)
Extend `options_json` accepted by `cactus_transcribe`, `cactus_stream_transcribe_start`, and the SDK wrappers to include a `custom_vocabulary` field:
```jsonc
{
"max_tokens": 448,
"custom_vocabulary": ["Cactus", "Ndubuaku", "HIPAA", "Omeprazole"],
"vocabulary_boost": 5.0 // optional, default ~5.0 logit bias
}
```
No new FFI functions are needed — the vocabulary list flows through the existing `options_json` parameter.
### Engine layer (`engine.h` / `model.h`)
Add a logit-bias hook that fires every decoder step (both `WhisperModel::decode_with_audio` and `MoonshineModel::decode_with_audio`):
1. At init time, tokenize each custom vocabulary entry using the model's `BPETokenizer` / `SPTokenizer` and store the resulting `token_id → boost` map.
2. Before sampling, add the boost value to the logits of every token that appears in any vocabulary entry's tokenization. This is the same pattern already used by `ToolCallConstrainer::get_bias()` and `suppress_tokens_` in `WhisperModel`.
### SDK surface (React Native, Flutter, Kotlin, Swift, Rust, Python)
Each SDK already forwards `options_json` to the C FFI, so they only need:
- Documentation updates showing the new `custom_vocabulary` key.
- Optionally, a typed helper (e.g., `TranscribeOptions.customVocabulary`).
## Implementation Sketch
1. **Parse** `custom_vocabulary` and `vocabulary_boost` from `options_json` inside `cactus_transcribe.cpp` (and `cactus_stream.cpp`), using the same lightweight JSON parsing already in `cactus_utils.h`.
2. **Tokenize** each vocabulary entry via the model's tokenizer. Build `std::unordered_map vocab_bias` mapping each sub-token → boost.
3. **Apply bias** in the decode loop. `WhisperModel` already applies `suppress_tokens_` (sets logits to `-INFINITY`). The custom vocabulary bias is the positive mirror: add `+boost` to the logit of each biased token before temperature scaling / softmax. This should be straightforward given the existing `decode_with_audio` signature already returns via logit manipulation.
4. **Stream transcription** (`cactus_stream_transcribe_start` / `_process`) should accept and persist the vocabulary bias for the lifetime of the stream session.
5. **Tests**: extend `tests/test_stt.cpp` with a case that passes custom vocabulary containing a rare word and asserts it appears in the output.
## Scope & Constraints
- **No model changes** — this is purely a decoding-time logit bias.
- **No new FFI functions** — everything goes through the existing `options_json`.
- Boost values should be clamped to a sane range (e.g., 0–20) to avoid degenerate outputs.
- Works for both Whisper and Moonshine model types.
- The existing `prompt` parameter continues to work alongside custom vocabulary (they are complementary).
## References
- `cactus/ffi/cactus_ffi.h:66-77` — `cactus_transcribe` signature
- `cactus/ffi/cactus_transcribe.cpp:248-286` — decoder loop where bias would be applied
- `cactus/models/model.h:405-667` — `WhisperModel` with existing `suppress_tokens_` logit manipulation
- `cactus/models/model.h:670-829` — `MoonshineModel`
- `cactus/engine/engine.h:395-484` — `ToolCallConstrainer` as prior art for token biasing
Contributor guide
Assessment
This issue has not been assessed yet.