[Feature]: expand AWQ experiments to low-bit INT and Hadamard combinations
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Feature Description
Extend the AWQ evaluation in #2285 with the following experiments:
- Evaluate AWQ with `W3A16`.
- Evaluate AWQ with `W2A16G64`.
- Evaluate AWQ combined with Hadamard rotation for `MXFP4` and `NVFP4`.
Compare against the corresponding RTN, optimized RTN, native AutoRound, and Hadamard-only baselines where applicable. Record the exact algorithm order and configuration used for each combined experiment.
### Motivation and Use Case
The results in #2285 show that AWQ can improve some quantization configurations. More data is needed to understand whether those benefits extend to more aggressive low-bit integer schemes and whether AWQ and Hadamard rotation are complementary for MXFP/NVFP quantization.
### Definition of Done
- [ ] Run AWQ experiments for `W3A16`.
- [ ] Run AWQ experiments for `W2A16G64`.
- [ ] Run AWQ + Hadamard experiments for `MXFP4`.
- [ ] Run AWQ + Hadamard experiments for `NVFP4`.
- [ ] Compare results with relevant RTN, optimized RTN, AutoRound, AWQ-only, and Hadamard-only baselines.
- [ ] Use consistent calibration and evaluation settings across comparisons.
- [ ] Report per-task accuracy, average accuracy, runtime, peak RAM, and peak VRAM.
- [ ] Document the commands, algorithm order, and key findings.
### Additional Context
Follow-up to the discussion in #2285:
https://github.com/intel/auto-round/pull/2285#discussion_r3920796834
Contributor guide
Research direction
Start by reading issue #2285 and the linked pull-request discussion to understand the existing AWQ evaluation setup and relevant baselines. Run the requested W3A16, W2A16G64, MXFP4, and NVFP4 experiments with consistent calibration and evaluation settings. Done means reporting per-task and average accuracy, runtime, peak RAM and VRAM, plus commands, algorithm order, configurations, and findings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100