huggingface / huggingface/candle
Clean the duplicated processor in the quantized example
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hi team! Thanks to the awesome work that bringing rust to the game!
I found that the usage of `LogitsProcessor` in the quantized example is not proper which makes the `chat` process iterating unnecessary resources with bad performance as results, people trying out the example may think it is caused by candle (like the performance of candle sucks comparing with llama.cpp XD, I did think so before reviewing the code carefully )
since we are using a loop here
https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L508
we don't have to inference all tokens again here
https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L575
instead, we can just move the LogitsProcessor out of the global interactive loop with extra cache of tokens including users' prompts
https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L559
This could be related to #1939 , the example is super slow from the second prompt
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in candle-examples/examples/quantized/main.rs around lines 508, 559, and 575, tracing the interactive chat loop and its LogitsProcessor usage. Check how prompts and generated tokens are cached, then verify that the second prompt no longer re-infers all prior tokens and that quantized chat performance improves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100