huggingface / huggingface/diffusers

Torchao SD3 int8wo

Open
#13,266 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

Exception when click Run, SD3 quantization

### Reproduction

- Mellon + ModularDiffuser setup (https://github.com/cubiq/Mellon/blob/main/modules/ModularDiffusers/README.md)
- uv pip install -U torch torchao
- python main.py

SD3 Text Encoder Loader: Dtype bfloat16, Quantization: TorchAO, Quant Type: int8wo

### Logs

```shell
20260314 21.08.15 [ERR] An unexpected error occurred during exec: cannot import name 'float8_dynamic_activation_float8_weight' from 'torchao.quantization' (/media/wasami/comfyui/Mellon/.venv/lib/python3.12/site-packages/torchao/quantization/__init__.py)
20260314 21.08.15 [ERR] Error executing node 50pz3Zy-IBsosDgI_t6gC (modules.Experiments.SD3TextEncodersLoader)
20260314 21.08.15 [ERR] Traceback (most recent call last):
File "/media/wasami/comfyui/Mellon/mellon/NodeBase.py", line 216, in __call__
output = getattr(self, self.CALLBACK)(**self.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/modules/Experiments/StableDiffusion3.py", line 190, in execute
t5_encoder = self.mm_exec(lambda: quantize(t5_encoder, quantization, **kwargs), quant_device, exclude=[t5_encoder])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/mellon/NodeBase.py", line 465, in mm_exec
return memory_manager.exec(func, device, models, exclude, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/memory_menager.py", line 188, in exec
raise e
File "/media/wasami/comfyui/Mellon/utils/memory_menager.py", line 172, in exec
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/modules/Experiments/StableDiffusion3.py", line 190, in
t5_encoder = self.mm_exec(lambda: quantize(t5_encoder, quantization, **kwargs), quant_device, exclude=[t5_encoder])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 52, in quantize
torchao(model, quant_type=torchao_quant_type)
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 70, in torchao
dtype = get_torchao_quant_method(quant_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 96, in get_torchao_quant_method
from torchao.quantization import (
ImportError: cannot import name 'float8_dynamic_activation_float8_weight' from 'torchao.quantization' (/media/wasami/comfyui/Mellon/.venv/lib/python3.12/site-packages/torchao/quantization/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/media/wasami/comfyui/Mellon/mellon/server.py", line 1015, in execute_node
self.node_cache[id](**args)
File "/media/wasami/comfyui/Mellon/mellon/NodeBase.py", line 220, in __call__
raise RuntimeError(f"Error executing {self.module_name}.{self.class_name}: {e}")
RuntimeError: Error executing modules.Experiments.SD3TextEncodersLoader: cannot import name 'float8_dynamic_activation_float8_weight' from 'torchao.quantization' (/media/wasami/comfyui/Mellon/.venv/lib/python3.12/site-packages/torchao/quantization/__init__.py)

20260314 21.08.15 [ERR] Error occurred in Traceback (most recent call last):
File "/media/wasami/comfyui/Mellon/mellon/NodeBase.py", line 216, in __call__
output = getattr(self, self.CALLBACK)(**self.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/modules/Experiments/StableDiffusion3.py", line 190, in execute
t5_encoder = self.mm_exec(lambda: quantize(t5_encoder, quantization, **kwargs), quant_device, exclude=[t5_encoder])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/mellon/NodeBase.py", line 465, in mm_exec
return memory_manager.exec(func, device, models, exclude, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/memory_menager.py", line 188, in exec
raise e
File "/media/wasami/comfyui/Mellon/utils/memory_menager.py", line 172, in exec
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/modules/Experiments/StableDiffusion3.py", line 190, in
t5_encoder = self.mm_exec(lambda: quantize(t5_encoder, quantization, **kwargs), quant_device, exclude=[t5_encoder])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 52, in quantize
torchao(model, quant_type=torchao_quant_type)
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 70, in torchao
dtype = get_torchao_quant_method(quant_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/utils/quantization.py", line 96, in get_torchao_quant_method
from torchao.quantization import (
ImportError: cannot import name 'float8_dynamic_activation_float8_weight' from 'torchao.quantization' (/media/wasami/comfyui/Mellon/.venv/lib/python3.12/site-packages/torchao/quantization/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/media/wasami/comfyui/Mellon/mellon/server.py", line 318, in _main_worker
result = await self.loop.run_in_executor(None, partial(task, *args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wasami/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/wasami/comfyui/Mellon/mellon/server.py", line 916, in execute_graph
self.execute_node(id, nodes[id], sid)
File "/media/wasami/comfyui/Mellon/mellon/server.py", line 1019, in execute_node
raise e
File "/media/wasami/comfyui/Mellon/mellon/server.py", line 1015, in execute_node
self.node_cache[id](**args)
```

### System Info

Ubuntu 24.04
RTX 5090

### Who can help?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in utils/quantization.py, especially get_torchao_quant_method and the torchao call at line 70, then trace the SD3 path in modules/Experiments/StableDiffusion3.py. Reproduce the failure with the Mellon setup and the SD3 Text Encoder Loader using TorchAO int8wo. Done means the reported ImportError no longer occurs for this configuration and the quantization path executes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.