Comfy-Org / Comfy-Org/ComfyUI

[Feature Request] INT4 per-row quantization — extends int8_tensorwise + convrot protocol

Open
#14,721 17 comments 1 reaction 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

在 ComfyUI 核心 ops.py 中增加 INT4 per-row weight-only 量化格式 注册,作为现有 int8_tensorwise 的向下扩展。格式复用 comfy_quant 元数据 + convrot(QuaRot)协议,唯一区别是 scale 从逐通道变为逐行([rows, 1]),weight 为 2×int4/字节 packed uint8。这样任何实现(插件、Triton kernel、硬件加速)都可以在同一协议下加载 INT4 权重。

English:

Add INT4 per-row weight-only quantization format registration to ops.py, as a downward extension of the existing int8_tensorwise format. Reuses the comfy_quant metadata + convrot (QuaRot) protocol. The only differences: scales go from per-channel to per-row ([rows, 1]), and weights are packed as 2×int4 per byte uint8. This way any implementation — plugin, Triton kernel, hardware accelerator — can load INT4 weights under the same protocol.

### Existing Solutions

在 ops.py 量化格式注册段新增 int4_per_row 格式,复用已有设施:

comfy_quant JSON 元数据:同结构,quant_format: "int4_per_row"
convrot + convrot_groupsize:完全不变
weight_scale:per-row float16,形状 [rows, 1] 代替 [1]
weight:packed uint8,rows × cols/2 字节
现有完整实现可参考:https://github.com/JWLHS/ComfyUI-WINT4-XPU-beta(纯 Python,22 个模型已验证)。Triton 加速版本约 100 行 kernel 即可替代反量化循环,等 XPU Triton 后端成熟后统一迁移。核心改动量约 30 行。

English:

Register a new int4_per_row format alongside existing quantization formats in ops.py, reusing:

comfy_quant JSON metadata: same structure, quant_format: "int4_per_row"
convrot + convrot_groupsize: unchanged
weight_scale: per-row float16, shape [rows, 1] instead of [1]
weight: packed uint8, rows × cols/2 bytes
Reference implementation: https://github.com/JWLHS/ComfyUI-WINT4-XPU-beta (pure Python, 22 models verified). A Triton-accelerated version would replace the dequant loop with a ~100-line fused kernel, once XPU Triton backend matures. Core change: ~30 lines.

### Other

中文:

纯 Python 实现,不绑定硬件(CPU/CUDA/XPU/MPS 均可运行)
22 个模型已量产验证:Wan2.1/2.2、LTX2.3、FLUX2、Krea2、Qwen、SCAIL2、Boogu 等,正常出图/出视频
支持多 LoRA 叠加(LoKr + ICLoRA bake-in),非量化层融合零推理开销
Triton 加速已规划:单 kernel 解包+GEMM 融合,一旦 XPU 后端到位即迁移
如果团队有兴趣我提 PR;没有也无妨,插件独立运行。感谢已有的 int8_tensorwise + convrot 协议让这个扩展变得极其简单
English:

Pure Python, no hardware lock-in (runs on CPU/CUDA/XPU/MPS)
22 models battle-tested: Wan2.1/2.2, LTX2.3, FLUX2, Krea2, Qwen, SCAIL2, Boogu — verified image/video output
Multi-LoRA stacking (LoKr + ICLoRA bake-in), zero inference overhead for non-quantized layers
Triton acceleration planned: single fused unpack+GEMM kernel, ready when XPU Triton backend lands
If the team is interested I'll submit a PR; if not, the plugin works standalone. Thanks to the existing int8_tensorwise + convrot protocol for making this a trivial extension

Contributor guide

Open the contributing guide

Research direction

Start in ops.py at the existing int8_tensorwise quantization-format registration and compare it with the linked ComfyUI-WINT4-XPU-beta reference implementation. Check how the proposed comfy_quant metadata, convrot settings, per-row scales, and packed weights fit the existing protocol. Done means the int4_per_row format is registered with the specified metadata and tensor shapes, with compatibility verified for the stated loading path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.