huggingface / huggingface/candle

Any tips to speed up quantized Whisper inference on Android?

Open
#1,048 24 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hello, running q80 quantized whisper on Android (Pixel 7) is taking around 15 seconds for 5 seconds of audio. Is there any way to speed this up that I might not be aware of or is it just because candle isn't as optimized as something like whisper.cpp yet? whisper.cpp took around 3 seconds or less if I remember correctly. Although this was with a q40 model. Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported q80 Whisper inference benchmark on a Pixel 7, using the issue's 5-second audio case and comparing the roughly 15-second result with the cited whisper.cpp comparison. Profile the Candle Android inference path to identify the limiting component; done means a measured, documented improvement or a clear explanation of the remaining gap.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, rust
Domain
machine-learning, mobile-dev, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.