[Feature Request] INT4 per-row quantization — extends int8_tensorwise + convrot protocol
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
在 ComfyUI 核心 ops.py 中增加 INT4 per-row weight-only 量化格式 注册,作为现有 int8_tensorwise 的向下扩展。格式复用 comfy_quant 元数据 + convrot(QuaRot)协议,唯一区别是 scale 从逐通道变为逐行([rows, 1]),weight 为 2×int4/字节 packed uint8。这样任何实现(插件、Triton kernel、硬件加速)都可以在同一协议下加载 INT4 权重。
English:
Add INT4 per-row weight-only quantization format registration to ops.py, as a downward extension of the existing int8_tensorwise format. Reuses the comfy_quant metadata + convrot (QuaRot) protocol. The only differences: scales go from per-channel to per-row ([rows, 1]), and weights are packed as 2×int4 per byte uint8. This way any implementation — plugin, Triton kernel, hardware accelerator — can load INT4 weights under the same protocol.
### Existing Solutions
在 ops.py 量化格式注册段新增 int4_per_row 格式,复用已有设施:
comfy_quant JSON 元数据:同结构,quant_format: "int4_per_row"
convrot + convrot_groupsize:完全不变
weight_scale:per-row float16,形状 [rows, 1] 代替 [1]
weight:packed uint8,rows × cols/2 字节
现有完整实现可参考:https://github.com/JWLHS/ComfyUI-WINT4-XPU-beta(纯 Python,22 个模型已验证)。Triton 加速版本约 100 行 kernel 即可替代反量化循环,等 XPU Triton 后端成熟后统一迁移。核心改动量约 30 行。
English:
Register a new int4_per_row format alongside existing quantization formats in ops.py, reusing:
comfy_quant JSON metadata: same structure, quant_format: "int4_per_row"
convrot + convrot_groupsize: unchanged
weight_scale: per-row float16, shape [rows, 1] instead of [1]
weight: packed uint8, rows × cols/2 bytes
Reference implementation: https://github.com/JWLHS/ComfyUI-WINT4-XPU-beta (pure Python, 22 models verified). A Triton-accelerated version would replace the dequant loop with a ~100-line fused kernel, once XPU Triton backend matures. Core change: ~30 lines.
### Other
中文:
纯 Python 实现,不绑定硬件(CPU/CUDA/XPU/MPS 均可运行)
22 个模型已量产验证:Wan2.1/2.2、LTX2.3、FLUX2、Krea2、Qwen、SCAIL2、Boogu 等,正常出图/出视频
支持多 LoRA 叠加(LoKr + ICLoRA bake-in),非量化层融合零推理开销
Triton 加速已规划:单 kernel 解包+GEMM 融合,一旦 XPU 后端到位即迁移
如果团队有兴趣我提 PR;没有也无妨,插件独立运行。感谢已有的 int8_tensorwise + convrot 协议让这个扩展变得极其简单
English:
Pure Python, no hardware lock-in (runs on CPU/CUDA/XPU/MPS)
22 models battle-tested: Wan2.1/2.2, LTX2.3, FLUX2, Krea2, Qwen, SCAIL2, Boogu — verified image/video output
Multi-LoRA stacking (LoKr + ICLoRA bake-in), zero inference overhead for non-quantized layers
Triton acceleration planned: single fused unpack+GEMM kernel, ready when XPU Triton backend lands
If the team is interested I'll submit a PR; if not, the plugin works standalone. Thanks to the existing int8_tensorwise + convrot protocol for making this a trivial extension
Contributor guide
Research direction
Start in ops.py at the existing int8_tensorwise quantization-format registration and compare it with the linked ComfyUI-WINT4-XPU-beta reference implementation. Check how the proposed comfy_quant metadata, convrot settings, per-row scales, and packed weights fit the existing protocol. Done means the int4_per_row format is registered with the specified metadata and tensor shapes, with compatibility verified for the stated loading path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100