microsoft / microsoft/VPTQ

a version incompatibility issue

Open
#196 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
681
Forks
53
PR merge metrics
No merged PRs in 30d

Description

Hello, dear author. Could you please update the code in the algorithm branch?
Currently, when using a model quantized with the algorithm branch and trying to load it with the latest version of VPTQ, an error occurs. It seems that the model is not compatible with the latest VPTQ code. The error is similar to:
INFO 05-22 02:56:39 [importing.py:53] Triton module has been replaced with a placeholder. INFO 05-22 02:56:39 [__init__.py:239] Automatically detected platform cuda. Successfully loaded VPTQ CUDA kernels. Replacing linear layers...: 1%|█▊ | 6/423 [00:00<00:00, 100663.30it/s] Traceback (most recent call last): File "/dataST/users/gexinning/llm_test/evlu_vptq.py", line 50, in <module> model = VQAutoModelQuantization.from_pretrained( File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/vptq/layers/model_base.py", line 122, in from_pretrained make_quant_linear( File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/vptq/layers/model_base.py", line 46, in make_quant_linear new_module = target_layer( TypeError: vptq.layers.vqlinear.VQuantLinear() got multiple values for keyword argument 'enable_proxy_error'
and
INFO 05-21 14:19:21 [importing.py:53] Triton module has been replaced with a placeholder. INFO 05-21 14:19:21 [__init__.py:239] Automatically detected platform cuda. Successfully loaded VPTQ CUDA kernels. Replacing linear layers...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 215/215 [00:00<00:00, 1789.69it/s] 2025-05-21 14:19:24,207 - accelerate.utils.modeling - WARNING - The model weights are not tied. Please use the tie_weightsmethod before using theinfer_auto_devicefunction. 2025-05-21 14:19:24,211 - accelerate.utils.modeling - WARNING - The model weights are not tied. Please use thetie_weightsmethod before using theinfer_auto_device function. Traceback (most recent call last): File "/dataST/users/gexinning/llm_test/evlu_vptq.py", line 50, in <module> model = VQAutoModelQuantization.from_pretrained( File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/vptq/layers/model_base.py", line 185, in from_pretrained model = accelerate.load_checkpoint_and_dispatch( File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/accelerate/big_modeling.py", line 642, in load_checkpoint_and_dispatch return dispatch_model( File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/accelerate/big_modeling.py", line 502, in dispatch_model model.to(device) File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3698, in to return super().to(*args, **kwargs) File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1343, in to return self._apply(convert) File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/torch/nn/modules/module.py", line 903, in _apply module._apply(fn) File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/torch/nn/modules/module.py", line 930, in _apply param_applied = fn(param) File "/dataST/users/gexinning/.conda/envs/ptq/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1336, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in vptq/layers/model_base.py at make_quant_linear and VQAutoModelQuantization.from_pretrained, then inspect the VQuantLinear loading path referenced by both traces. Reproduce loading a model quantized on the algorithm branch with the latest VPTQ code and compare the two reported failures. Done means the model loads without either the duplicate keyword or meta-tensor error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.