How to dequantize a model with 4 groups and centroids greater than 4096?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 681
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
I've been trying to quantize and run the Meta-Llama-3.1-8B-Instruct-2.3bit model with group number set to 4, and successfully run the model when k1(centroids) is 4096 as in the paper. However, anything k1 setting above that(8192, 16384, 65536) would lead to a successful quantization but a model running failure. The error logs show that the reason could be some illegal memory access during the dequant function.
So here's what I want to ask, does the code support running a model with group number option on and centroids set to 8k and greater? Or do I need to do some adjustment to make it work?
Looking forward to your reply.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dequant function and the error logs from running the quantized Meta-Llama-3.1-8B-Instruct-2.3bit model. Reproduce the failure with group number 4 and k1 values of 8192, 16384, and 65536, comparing them with 4096. Done means establishing whether these centroid sizes are supported and identifying the cause of the illegal memory access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100