deepspeedai / deepspeedai/DeepSpeedExamples
[inference] when will deepspeed support inference with int8 gemm?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
For now, deepspeed inference with int8 model is still use fp16 cublas gemm function and the quantized weights(int8) are dequantized back to fp16 as shown here:
When will it support real int8 inference which uses the int8 data to do gemm ? In this case, the activation or hidden states are also quantized to int8 data type which can reduce the memory requirements dramatically
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at csrc/transformer/inference/csrc/pt_binding.cpp around line 863, where the issue identifies the current inference behavior. Determine the scope needed for real int8 GEMM using quantized weights and activations without dequantizing weights to fp16; the issue does not name tests or a specific implementation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100