How to Generate a 2-bit Quantized Meta-Llama-3.1-8B-Instruct Model?
@YangWang92 is already working on this.
Since Nov 21, 2024.
- Dominant language
- Python
- Stars
- 681
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
I found a similar closed issue related to this topic. Following your reply in that issue, I successfully configured the vptq-algo environment based on the tutorial in the algorithm branch.
The Quantization on Meta-Llama-3.1-8B-Instruct section provides an example of using VPTQ to generate 3-bit quantized Meta-Llama-3.1-8B-Instruct model. However, if I want to generate the 2.3-bit quantized Meta-Llama-3.1-8B-Instruct model provided in VPTQ-community, how should I configure the parameters for run_vptq.py? Specifically, which arguments should I adjust to achieve 2.3-bit quantization?
Looking forward to your reply.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.