intel / intel/auto-round

[Feature]: All the algorithm code should be decoupled from datatype and format-specific behavior

Open
#1,802 1 comment 0 reactions 1 assignee Claimed by @n1ck-guo View on GitHub
enhancement high priority
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

### Feature Description

It is a significant burden for algorithm developers to handle format- or datatype-specific logic inside the algorithm implementation itself, e.g., update_block_global_scale_if_needed.

code in current opt-rtn
~~~
def quantize_block(
self, block: torch.nn.Module, input_ids=None, input_others=None, reference_output=None, **kwargs
):
"""Apply imatrix-informed RTN quantization to a block.

Pure-algorithm entry point. All infrastructure (device placement,
act-max hook registration, imatrix collection, cleanup) is handled
by the Compressor before calling this method.

Args:
block: Module already placed on the correct device(s) with act_max
attributes populated by the Compressor's hook pass.
input_ids: Unused for optimized RTN; accepted for interface consistency.
input_others: Unused for optimized RTN.
reference_output: Unused for optimized RTN.
"""
update_block_global_scale_if_needed(block, self.data_type, self.group_size)
if (
self.config.is_act_nv_fp
or self.config.is_static_afp8
or (self.config.is_wfp8afp8 and not self.config.act_dynamic)
):
# enable moe experts act_max automatic generation for Linear
set_amax_for_all_moe_layers(block, attr_name="act_max")
# Normalize imatrix and quantize layers
for name, m in block.named_modules():
if hasattr(m, "imatrix"):
m.imatrix /= m.imatrix_cnt
if hasattr(m, "global_name") and check_to_quantized(m):
self.quantize_layer(m.global_name)
~~~

### Motivation and Use Case

~

### Alternatives Considered

~

### Definition of Done

_No response_

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.