deepseek-ai / deepseek-ai/DeepSeek-Coder

TensorRT Quantization Breaks for `LlamaLinearScalingRotaryEmbedding`

Open
#117 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24.3k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

In nvidia-ammo, it appears these lines in `ammo/torch/export/layer_utils.py` have an unexpected failure for some Llama variants:

Screen Shot 2024-02-10 at 11 12 23 PM

In particular, the deepseek models use `LlamaLinearScalingRotaryEmbedding`. This means the module is picked up by the `is_linear` check, and is treated as the dense case. However, there is no `.weight` for this module, so the `build_linear_config` fails.

Lots of easy fixes for this (for example, just checking if "Rotary" in name and skipping that case), happy to contribute (but don't think there is an OSS repo to do so)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ammo/torch/export/layer_utils.py and inspect the is_linear check and the build_linear_config path for LlamaLinearScalingRotaryEmbedding. Reproduce quantization with an affected DeepSeek or Llama variant, then verify that the rotary module no longer causes the missing-weight failure and that the affected export completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.