huggingface / huggingface/candle

Clean the duplicated processor in the quantized example

Open
#2,705 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hi team! Thanks to the awesome work that bringing rust to the game!

I found that the usage of `LogitsProcessor` in the quantized example is not proper which makes the `chat` process iterating unnecessary resources with bad performance as results, people trying out the example may think it is caused by candle (like the performance of candle sucks comparing with llama.cpp XD, I did think so before reviewing the code carefully )

since we are using a loop here

https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L508

we don't have to inference all tokens again here

https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L575

instead, we can just move the LogitsProcessor out of the global interactive loop with extra cache of tokens including users' prompts

https://github.com/huggingface/candle/blob/236c35e5789723efe772f41920f3ac071bdff24d/candle-examples/examples/quantized/main.rs#L559

This could be related to #1939 , the example is super slow from the second prompt

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-examples/examples/quantized/main.rs around lines 508, 559, and 575, tracing the interactive chat loop and its LogitsProcessor usage. Check how prompts and generated tokens are cached, then verify that the second prompt no longer re-infers all prior tokens and that quantized chat performance improves.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.