Comfy-Org / Comfy-Org/ComfyUI

Conv3D with bfloat16 raises Internal assert failed error on Ascend

Open
#7,784 0 comments 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Expected Behavior

Use the official workflow When using nn.Conv3d with bfloat16 input on Ascend 910B, I encounter the following internal assertion error during runtime:

### Actual Behavior

The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv3D.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-25-00:49:58 (PID:755653, Device:0, RankID:-1) ERR00100 PTA call acl api failed![Image](https://github.com/user-attachments/assets/848fa7d4-8e92-40c8-b281-bdaebf2dc2d5)

### Steps to Reproduce

Use the models and video workflow provided by the official repository [](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/diffusion_models)

workflow :[[](url)](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/example%20workflows_Wan2.1/text_to_video_wan.json)

### Debug Logs

```powershell
got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: npu:0, offload device: cpu, dtype: torch.bfloat16
CLIP/text encoder model load device: npu:0, offload device: cpu, current: cpu, dtype: torch.float16
FETCH ComfyRegistry Data: 60/82
Requested to load WanTEModel
FETCH ComfyRegistry Data: 65/82
FETCH ComfyRegistry Data: 70/82
FETCH ComfyRegistry Data: 75/82
loaded completely 48031.92890625 10835.4765625 True
model weight dtype torch.float16, manual cast: None
model_type FLOW
Requested to load WAN21
FETCH ComfyRegistry Data: 80/82
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.jsonloaded completely 37135.68952890625 2706.1788330078125 True

0%| | 0/10 [00:00
lambda input, sigma, **kwargs: predict_eps_sigma(model, input, sigma, **kwargs),
File "/apps/ComfyUI/comfy/extra_samplers/uni_pc.py", line 843, in predict_eps_sigma
return (input - model(input, sigma_in, **kwargs)) / sigma
File "/apps/ComfyUI/comfy/samplers.py", line 390, in __call__
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/apps/ComfyUI/comfy/samplers.py", line 939, in __call__
return self.predict_noise(*args, **kwargs)
File "/apps/ComfyUI/comfy/samplers.py", line 942, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/apps/ComfyUI/comfy/samplers.py", line 370, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
File "/apps/ComfyUI/comfy/samplers.py", line 206, in calc_cond_batch
return executor.execute(model, conds, x_in, timestep, model_options)
File "/apps/ComfyUI/comfy/patcher_extension.py", line 111, in execute
return self.original(*args, **kwargs)
File "/apps/ComfyUI/comfy/samplers.py", line 319, in _calc_cond_batch
output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
File "/apps/ComfyUI/comfy/model_base.py", line 138, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
File "/apps/ComfyUI/comfy/patcher_extension.py", line 111, in execute
return self.original(*args, **kwargs)
File "/apps/ComfyUI/comfy/model_base.py", line 171, in _apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/data/anaconda3/envs/comfyui_3.10.x/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/comfyui_3.10.x/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/apps/ComfyUI/comfy/ldm/wan/model.py", line 474, in forward
return self.forward_orig(x, timestep, context, clip_fea=clip_fea, freqs=freqs, transformer_options=transformer_options)[:, :, :t, :h, :w]
File "/apps/ComfyUI/comfy/ldm/wan/model.py", line 427, in forward_orig
sinusoidal_embedding_1d(self.freq_dim, t).to(dtype=x[0].dtype))
File "/apps/ComfyUI/comfy/ldm/wan/model.py", line 25, in sinusoidal_embedding_1d
position, torch.pow(10000, -torch.arange(half).to(position).div(half)))
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv3D.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-25-00:26:06 (PID:743850, Device:0, RankID:-1) ERR00100 PTA call acl api failed
```

### Other

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.