[Feature]: All the algorithm code should be decoupled from datatype and format-specific behavior
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 175
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 99
Description
### Feature Description
It is a significant burden for algorithm developers to handle format- or datatype-specific logic inside the algorithm implementation itself, e.g., update_block_global_scale_if_needed.
code in current opt-rtn
~~~
def quantize_block(
self, block: torch.nn.Module, input_ids=None, input_others=None, reference_output=None, **kwargs
):
"""Apply imatrix-informed RTN quantization to a block.
Pure-algorithm entry point. All infrastructure (device placement,
act-max hook registration, imatrix collection, cleanup) is handled
by the Compressor before calling this method.
Args:
block: Module already placed on the correct device(s) with act_max
attributes populated by the Compressor's hook pass.
input_ids: Unused for optimized RTN; accepted for interface consistency.
input_others: Unused for optimized RTN.
reference_output: Unused for optimized RTN.
"""
update_block_global_scale_if_needed(block, self.data_type, self.group_size)
if (
self.config.is_act_nv_fp
or self.config.is_static_afp8
or (self.config.is_wfp8afp8 and not self.config.act_dynamic)
):
# enable moe experts act_max automatic generation for Linear
set_amax_for_all_moe_layers(block, attr_name="act_max")
# Normalize imatrix and quantize layers
for name, m in block.named_modules():
if hasattr(m, "imatrix"):
m.imatrix /= m.imatrix_cnt
if hasattr(m, "global_name") and check_to_quantized(m):
self.quantize_layer(m.global_name)
~~~
### Motivation and Use Case
~
### Alternatives Considered
~
### Definition of Done
_No response_
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.