huggingface / huggingface/candle
Any tips to speed up quantized Whisper inference on Android?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hello, running q80 quantized whisper on Android (Pixel 7) is taking around 15 seconds for 5 seconds of audio. Is there any way to speed this up that I might not be aware of or is it just because candle isn't as optimized as something like whisper.cpp yet? whisper.cpp took around 3 seconds or less if I remember correctly. Although this was with a q40 model. Thanks.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported q80 Whisper inference benchmark on a Pixel 7, using the issue's 5-second audio case and comparing the roughly 15-second result with the cited whisper.cpp comparison. Profile the Candle Android inference path to identify the limiting component; done means a measured, documented improvement or a clear explanation of the remaining gap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, rust
- Domain
- machine-learning, mobile-dev, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100