Comfy-Org / Comfy-Org/comfy-kitchen
Bug: Scales do not persist in fp32 when LoRA is applied.
- Dominant language
- Python
- Stars
- 220
- Forks
- 91
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 12
Description
I am on ops-changes branch of ComfyUI.
There was no issues when no LoRA was being Loaded.
Model loaded: https://huggingface.co/silveroxides/Chroma1-HD-fp8-scaled/blob/main/Chroma1-HD-fp8matmulmixed_large_rev2.safetensors
LoRA used: https://huggingface.co/silveroxides/Chroma-LoRAs/blob/main/flash-heun-pruned/chroma-flash-heun_r01-fp32-pruned.safetensors
workflow: [comfy-kitchen-test.json](https://github.com/user-attachments/files/24440488/comfy-kitchen-test.json)
```
FP8 _scaled_mm failed: Invalid scaling configuration.
- For TensorWise scaling, a and b should be float8, scales should be float and singletons.
- For RowWise scaling, a and b should be float8, scales should be float, scale_a should be (7056, 1) and scale_b should be (1, 9216), and both should be contiguous.
- For BlockWise 1x128 scaling, a and b should be float8, scales should be float, scale_a should be (7056, 24) and scale_b should be (24, 9216), and both should be outer-dim-major.
- For BlockWise 128x128 scaling, a and b should be float8, scales should be float, scale_a should be (56, 24) and scale_b should be (24, 72), and both should be near-inner-dim-major (with 16-byte aligned strides).
- For Blockwise 1x32 scaling, a and b should be float8, scales should be float8_e8m0fnu, scale_a should have 688128 elements and scale_b should have 884736 elements, and both should be contiguous.
- For Blockwise 1x16 scaling, a and b should be float4 (packed 2x), scales should be float8_e4m3fn, scale_a should have 2752512 elements and scale_b should have 3538944 elements, and both should be contiguous.
Got a.dtype()=Float8_e4m3fn, scale_a.dtype()=Float, scale_a.size()=[1], scale_a.stride()=[1], b.dtype()=Float8_e4m3fn, scale_b.dtype()=Half, scale_b.size()=[] and scale_b.stride()=[], falling back to dequantization
0%| | 0/26 [00:00
return type(args)(dequantize_args(a) for a in args)
^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\tensor\base.py", line 347, in dequantize_args
return args.dequantize()
^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\tensor\base.py", line 265, in dequantize
full = self._layout_cls.dequantize(qdata, self._params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\tensor\fp8.py", line 67, in dequantize
return ck.dequantize_per_tensor_fp8(qdata, params.scale, params.orig_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\__init__.py", line 84, in dequantize_per_tensor_fp8
return torch.ops.comfy_kitchen.dequantize_fp8(x, scale, dtype_code)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\torch\_ops.py", line 1255, in __call__
return self._op(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\torch\_library\custom_ops.py", line 343, in backend_impl
result = self._backend_fns[device_type](*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\torch\_compile.py", line 53, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 1044, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\torch\_library\custom_ops.py", line 376, in wrapped_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\backends\eager\quantization.py", line 241, in _op_dequantize_fp8
impl = registry.get_implementation("dequantize_per_tensor_fp8", kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\registry.py", line 269, in get_implementation
selected_backend = self.get_capable_backend(func_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ishim\Tools\ComfyUI\venv\Lib\site-packages\comfy_kitchen\registry.py", line 233, in get_capable_backend
raise NoCapableBackendError(func_name, failures)
comfy_kitchen.exceptions.NoCapableBackendError: No backend can handle 'dequantize_per_tensor_fp8': triton: scale: dtype torch.float16 not in {torch.float32}; eager: scale: dtype torch.float16 not in {torch.float32}
```
Contributor guide
Assessment
This issue has not been assessed yet.