deepseek-ai / deepseek-ai/DeepSeek-Coder
TensorRT Quantization Breaks for `LlamaLinearScalingRotaryEmbedding`
- Dominant language
- Python
- Stars
- 24.3k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
In nvidia-ammo, it appears these lines in `ammo/torch/export/layer_utils.py` have an unexpected failure for some Llama variants:
In particular, the deepseek models use `LlamaLinearScalingRotaryEmbedding`. This means the module is picked up by the `is_linear` check, and is treated as the dense case. However, there is no `.weight` for this module, so the `build_linear_config` fails.
Lots of easy fixes for this (for example, just checking if "Rotary" in name and skipping that case), happy to contribute (but don't think there is an OSS repo to do so)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in ammo/torch/export/layer_utils.py and inspect the is_linear check and the build_linear_config path for LlamaLinearScalingRotaryEmbedding. Reproduce quantization with an affected DeepSeek or Llama variant, then verify that the rotary module no longer causes the missing-weight failure and that the affected export completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100