intel / intel/auto-round

Support Nemotron-H / Nemotron-Cascade-2 (hybrid Mamba2 + Attention + MoE)

Open
#1,711 0 comments 1 reaction 1 assignee Claimed by @lkk12014402 View on GitHub
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

## Goal

Enable AutoRound to quantize Nemotron-H family models — in particular `nvidia/Nemotron-Cascade-2-30B-A3B` — and produce a checkpoint whose decoded output is coherent without requiring launcher-side workarounds.

Target user flow:

```python
from auto_round import AutoRound
AutoRound("nvidia/Nemotron-Cascade-2-30B-A3B", bits=4, group_size=128, sym=True) \
.quantize_and_save(output_dir=..., format="auto_round")
```

## Scope

- Nemotron-H architecture (hybrid Mamba2 SSM + Attention + routed MoE + shared expert)
- `auto_round` export format, W4A16 weight-only
- CPU + CUDA/XPU calibration paths

## Acceptance

- Bare `AutoRound("nvidia/...")` call produces a checkpoint that loads and generates coherent text in a downstream inference engine.
- No regressions for other `model_type` values (non-NH models must be unaffected).
- CPU test suite covers registration, post-load behaviour, and export-time precision controls.

## Out of scope (follow-ups)

- Pure-Mamba architectures (no attention layers)
- Other Nemotron-H variants beyond the Cascade-2 checkpoint family

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.