[Feature]: very slow to quatnize qwen3.8-flash-next for full attention layers

with this pr https://github.com/intel/auto-round/pull/2303
### 1 cards
(wenhuach) wenhuach@sc09dell06-nvd:~/auto-round-moe$ CUDA_VISIBLE_DEVICES=7 python3 -m auto_round ../Qwen3.8-Flash-Next/ --device_map auto
2026-09-08 21:55:03 INFO main.py L310: start to quantize ../Qwen3.8-Flash-Next
2026-09-08 21:55:03 INFO model.py L332: Fused-MoE checkpoint detected: building a meta skeleton and materializing weights per block (set `AR_DISABLE_AUTO_META_LOAD=1` to load the whole model on CPU instead).
2026-09-08 21:55:06 INFO device.py L1558: Before applying custom replacements 'peak_ram': 0.97GB
2026-09-08 21:55:09 INFO moe_experts_interface.py L793: [MoE Prep] Unfused 48 MOE experts modules
2026-09-08 21:55:10 INFO device.py L1558: After applying custom replacements 'peak_ram': 1.32GB
2026-09-08 21:55:10 INFO replace_modules.py L101: Prepared 48 MOE modules for quantization
2026-09-08 21:55:10 INFO replace_modules.py L160: Experts (before unfuse) [model.language_model.layers.0.mlp.experts] (Qwen4ExpTextExperts):
Qwen4ExpTextExperts(
(act_fn): SiLUActivation()
)
2026-09-08 21:55:10 INFO replace_modules.py L161: Experts (after unfuse) [model.language_model.layers.0.mlp.experts] (Qwen4ExpTextExperts):
Qwen4ExpTextExperts(
(act_fn): SiLUActivation()
(0-511): 512 x _ExpertContainer(
(down_proj): Linear(in_features=640, out_features=2560, bias=False)
(gate_proj): Linear(in_features=2560, out_features=640, bias=False)
(up_proj): Linear(in_features=2560, out_features=640, bias=False)
)
)
2026-09-08 21:55:13 INFO base.py L2138: format is not set, using default auto_round format.
2026-09-08 21:55:13 WARNING logging.py L340: some layers are skipped quantization (shape not divisible by 32): model.language_model.layers.[0-2,4-6,8-10,12-14,16-18,20-22,24-26,28-30,32-34,36-38,40-42,44-46].linear_attn.in_proj_a, model.language_model.layers.[0-2,4-6,8-10,12-14,16-18,20-22,24-26,28-30,32-34,36-38,40-42,44-46].linear_attn.in_proj_b, model.language_model.layers.[0-47].attn_hyper_connection.block_inject_weight, model.language_model.layers.[0-47].mlp.shared_expert_gate, model.language_model.layers.[0-47].mlp_hyper_connection.block_inject_weight, model.visual.blocks.[0-26].mlp.linear_fc1, model.visual.blocks.[0-26].mlp.linear_fc2
2026-09-08 21:55:13 INFO base.py L999: Using predefined ignore_layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 21:55:13 INFO layer_config_resolver.py L401: Ignored layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 21:55:16 INFO layer_config_resolver.py L401: Ignored layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 21:55:22 INFO base.py L1340: `torch.compile` is enabled
qqq2026-09-08 21:55:36 INFO orchestrator.py L587: start to cache block inputs
2026-09-08 21:55:36 INFO mllm.py L86: Using MLLM template: qwen4_exp
2026-09-08 21:55:36 INFO mllm.py L125: Multimodal model with non-MLLM calibration dataset 'NeelNanda/pile-10k' and quant_nontext_module=False: using the standard text dataloader (vision/audio towers are not being quantized, so text-only calibration through the full-model forward is sufficient).
2026-09-08 21:55:36 INFO calib_dataset.py L1113: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
2026-09-08 21:55:41 INFO device.py L1560: 'peak_ram': 7.85GB
2026-09-08 21:55:41 INFO orchestrator.py L619: caching done
Quantizing model.language_model.layers.0: 0%| | 0/48 [00:03 iter 199: 5.740e-07
2026-09-08 21:56:46 INFO device.py L1560: 'peak_ram': 20.75GB, 'peak_vram': 81.9GB
Quantizing model.language_model.layers.1: 2%|███▋ | 1/48 [02:39<2:05:13, 159.86s/it]2026-09-08 21:58:34 INFO logging.py L355: Detected 1 ngram embedding module(s), total size 95.37 GiB (AR_NGRAM_DEVICE=auto).
2026-09-08 21:58:34 INFO logging.py L355: AR_NGRAM_DEVICE is not set, so ngram embeddings stay on CPU by default (total 95.37 GiB, memory-safe). For faster lookup set AR_NGRAM_DEVICE= (single GPU) or AR_NGRAM_DEVICE=across (multi-GPU sharding, experimental and may have bugs).
2026-09-08 21:58:34 INFO model.py L2962: Placed 1 ngram embedding module(s) on cpu (total 95.37 GiB).
2026-09-08 21:59:36 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1548/1553 layers in the block, loss iter 0: 1.808e-06 -> iter 128: 1.021e-06
2026-09-08 21:59:44 INFO device.py L1560: 'peak_ram': 117.69GB, 'peak_vram': 92.46GB
Quantizing model.language_model.layers.2: 4%|███████▎ | 2/48 [10:59<4:35:36, 359.48s/it]2026-09-08 22:07:27 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 2.918e-06 -> iter 122: 1.301e-06
2026-09-08 22:07:33 INFO device.py L1560: 'peak_ram': 117.69GB, 'peak_vram': 92.46GB
Quantizing model.language_model.layers.3: 6%|██████████▉ | 3/48 [13:36<3:20:17, 267.06s/it]2026-09-08 22:40:58 INFO quantizer.py L550: Unquantized layers: ['mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1548/1551 layers in the block, loss iter 0: 2.992e-06 -> iter 186: 1.592e-06
2026-09-08 22:42:48 INFO device.py L1560: 'peak_ram': 117.69GB, 'peak_vram': 92.46GB
Quantizing model.language_model.layers.4: 8%|██████████████▍ | 4/48 [48:58<12:13:01, 999.59s/it]2026-09-08 22:45:25 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 4.482e-06 -> iter 158: 1.982e-06
2026-09-08 22:45:32 INFO device.py L1560: 'peak_ram': 117.69GB, 'peak_vram': 92.46GB
Quantizing model.language_model.layers.5: 10%|██████████████████▏ | 5/48 [51:28<8:16:48, 693.22s/it]2026-09-08 22:47:55 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 4.841e-06 -> iter 165: 2.336e-06
2026-09-08 22:48:01 INFO device.py L1560: 'peak_ram': 117.69GB, 'peak_vram': 92.46GB
Quantizing model.language_model.layers.6: 12%|█████████████████████▊ | 6/48 [54:11<5:58:59, 512.84s/it]

### 2 cards
~/auto-round-moe$ AR_NGRAM_DEVICE=across CUDA_VISIBLE_DEVICES=6,7 python3 -m auto_round ../Qwen3.8-Flash-Next/ --device_map auto
2026-09-08 20:21:30 INFO main.py L310: start to quantize ../Qwen3.8-Flash-Next
2026-09-08 20:21:30 INFO model.py L332: Fused-MoE checkpoint detected: building a meta skeleton and materializing weights per block (set `AR_DISABLE_AUTO_META_LOAD=1` to load the whole model on CPU instead).
2026-09-08 20:21:33 INFO device.py L1558: Before applying custom replacements 'peak_ram': 0.97GB
2026-09-08 20:21:36 INFO moe_experts_interface.py L793: [MoE Prep] Unfused 48 MOE experts modules
2026-09-08 20:21:37 INFO device.py L1558: After applying custom replacements 'peak_ram': 1.41GB
2026-09-08 20:21:37 INFO replace_modules.py L101: Prepared 48 MOE modules for quantization
2026-09-08 20:21:37 INFO replace_modules.py L160: Experts (before unfuse) [model.language_model.layers.0.mlp.experts] (Qwen4ExpTextExperts):
Qwen4ExpTextExperts(
(act_fn): SiLUActivation()
)
2026-09-08 20:21:37 INFO replace_modules.py L161: Experts (after unfuse) [model.language_model.layers.0.mlp.experts] (Qwen4ExpTextExperts):
Qwen4ExpTextExperts(
(act_fn): SiLUActivation()
(0-511): 512 x _ExpertContainer(
(down_proj): Linear(in_features=640, out_features=2560, bias=False)
(gate_proj): Linear(in_features=2560, out_features=640, bias=False)
(up_proj): Linear(in_features=2560, out_features=640, bias=False)
)
)
2026-09-08 20:21:40 INFO base.py L2138: format is not set, using default auto_round format.
2026-09-08 20:21:40 WARNING logging.py L340: some layers are skipped quantization (shape not divisible by 32): model.language_model.layers.[0-2,4-6,8-10,12-14,16-18,20-22,24-26,28-30,32-34,36-38,40-42,44-46].linear_attn.in_proj_a, model.language_model.layers.[0-2,4-6,8-10,12-14,16-18,20-22,24-26,28-30,32-34,36-38,40-42,44-46].linear_attn.in_proj_b, model.language_model.layers.[0-47].attn_hyper_connection.block_inject_weight, model.language_model.layers.[0-47].mlp.shared_expert_gate, model.language_model.layers.[0-47].mlp_hyper_connection.block_inject_weight, model.visual.blocks.[0-26].mlp.linear_fc1, model.visual.blocks.[0-26].mlp.linear_fc2
2026-09-08 20:21:40 INFO base.py L999: Using predefined ignore_layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 20:21:40 INFO layer_config_resolver.py L401: Ignored layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 20:21:43 INFO layer_config_resolver.py L401: Ignored layers: model.language_model.layers.[0-47].mlp.gate
2026-09-08 20:21:49 INFO base.py L1340: `torch.compile` is enabled
2026-09-08 20:22:02 INFO orchestrator.py L587: start to cache block inputs
2026-09-08 20:22:03 INFO mllm.py L86: Using MLLM template: qwen4_exp
2026-09-08 20:22:03 INFO mllm.py L125: Multimodal model with non-MLLM calibration dataset 'NeelNanda/pile-10k' and quant_nontext_module=False: using the standard text dataloader (vision/audio towers are not being quantized, so text-only calibration through the full-model forward is sufficient).
2026-09-08 20:22:03 INFO calib_dataset.py L1113: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
2026-09-08 20:22:07 INFO device.py L1560: 'peak_ram': 7.94GB
2026-09-08 20:22:07 INFO orchestrator.py L619: caching done
Quantizing model.language_model.layers.0: 0%| | 0/48 [00:03 iter 199: 5.603e-07
2026-09-08 20:24:19 INFO device.py L1560: 'peak_ram': 20.91GB, 'peak_vram': {'0': 40.24GB, '1': 47.83GB}
Quantizing model.language_model.layers.1: 2%|███▋ | 1/48 [03:53<3:02:42, 233.25s/it]2026-09-08 20:26:13 INFO logging.py L355: Detected 1 ngram embedding module(s), total size 95.37 GiB (AR_NGRAM_DEVICE=across).
2026-09-08 20:26:13 WARNING logging.py L340: Sharding ngram embeddings across multiple GPUs is experimental and may have bugs; set AR_NGRAM_DEVICE=cpu (safe) or a specific card (e.g. AR_NGRAM_DEVICE=cuda:0) if you hit issues.
2026-09-08 20:26:25 INFO model.py L2934: Placed 1 ngram embedding module(s) across 2 GPUs (total 95.37 GiB).
2026-09-08 20:28:51 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1548/1553 layers in the block, loss iter 0: 1.777e-06 -> iter 128: 9.988e-07
2026-09-08 20:30:37 INFO device.py L1560: 'peak_ram': 117.85GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.2: 4%|███████▎ | 2/48 [10:20<4:08:18, 323.88s/it]2026-09-08 20:34:28 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 2.887e-06 -> iter 122: 1.288e-06
2026-09-08 20:34:36 INFO device.py L1560: 'peak_ram': 119.4GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.3: 6%|██████████▉ | 3/48 [14:19<3:33:48, 285.07s/it]2026-09-08 21:06:43 INFO quantizer.py L550: Unquantized layers: ['mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1548/1551 layers in the block, loss iter 0: 2.998e-06 -> iter 186: 1.576e-06
2026-09-08 21:08:18 INFO device.py L1560: 'peak_ram': 121.26GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.4: 8%|██████████████▍ | 4/48 [48:34<12:01:23, 983.72s/it]2026-09-08 21:12:42 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 4.575e-06 -> iter 158: 1.971e-06
2026-09-08 21:12:51 INFO device.py L1560: 'peak_ram': 121.26GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.5: 10%|██████████████████▏ | 5/48 [52:24<8:30:17, 712.04s/it]2026-09-08 21:16:33 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 4.787e-06 -> iter 165: 2.311e-06
2026-09-08 21:16:41 INFO device.py L1560: 'peak_ram': 121.26GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.6: 12%|█████████████████████▊ | 6/48 [56:12<6:23:10, 547.39s/it]2026-09-08 21:20:21 INFO quantizer.py L550: Unquantized layers: ['linear_attn.in_proj_b', 'linear_attn.in_proj_a', 'mlp.shared_expert_gate', 'attn_hyper_connection.block_inject_weight', 'mlp_hyper_connection.block_inject_weight']
quantized 1546/1551 layers in the block, loss iter 0: 4.715e-06 -> iter 131: 2.608e-06
2026-09-08 21:20:32 INFO device.py L1560: 'peak_ram': 121.26GB, 'peak_vram': {'0': 92.73GB, '1': 100.31GB}
Quantizing model.language_model.layers.7: 15%|█████████████████████████ | 7/48 [1:00:06<5:04:02, 444.95s/it]

### Motivation and Use Case

~

### Alternatives Considered

_No response_

### Definition of Done

_No response_

### Additional Context

_No response_

Reproduce the commands in the issue through the auto_round entry point, using the Qwen3.8-Flash-Next model and the one- and two-GPU configurations shown. Start by tracing the block quantization and ngram-device paths mentioned in the logs, including quantizer.py, orchestrator.py, and model.py. Done means identifying and addressing the cause of the excessive per-layer runtime while preserving successful quantization.

Written by the indexing model from the issue text.