huggingface / huggingface/candle
Does candle gptq int8 faster than pytorch on CPU?
Open
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
As there nothing speed compare with int8 speed with pytorch vanilla (normally with transformers), does candle faster than pytorch on int8 in CPU than pytorch? By how much?
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating Candle's GPTQ int8 CPU path and a comparable vanilla PyTorch/Transformers setup, then run equivalent CPU benchmarks and document the measured speed difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch, rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100