Comfy-Org / Comfy-Org/ComfyUI

Triton pointer error on subsequent workflow runs or when generating with batch count >1

Open
#6,917 16 comments 3 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 10h
Merged PRs (30d)
153

Description

### Expected Behavior

The workflow runs successfully on subequent runs if the first run was successful.

### Actual Behavior

The workflow runs successfully on the first execution after the server is started, and then errors out on any subsequent run (or on subsequent batches when batch count >1), until the server is restarted.

### Steps to Reproduce

I've been trying to optimize Hunyuan inference on my gaming desktop, and managed to achieve ~18 minute end-to-end inference at the model's maximum trained resolution/length of 1280x720x129f, using a combination of fp8 fast mode, sage attention, torch compile, and `--lowvram` CPU offloading (necessary to avoid OOM in this case with 24gb vram).

The bad news is that it only works *once* per server restart. Any subsequent run results in `Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)` until the server is restarted.

My hypothesis is that this has something to do with the combination of torch compile with `--lowvram` blockswap CPU offloading, as this issue does not seem to occur if I reduce the generation resolution/frames so that CPU offloading is not required (such as generating a bunch of 1-frame still images). Perhaps some tensors are ending up on CPU at the end of the run and some kind of cleanup is needed prior to the next run?

My setup:
Windows 11
Python 3.11, Torch 2.6.0+cu124, Triton 3.2.0, Sage Attention 2.1.0
Ryzen 7950X, 128gb ram
RTX 4090 24gb vram
Launch args --lowvram --use-sage-attention

Workflow:
```
{"last_node_id":79,"last_link_id":224,"nodes":[{"id":26,"type":"FluxGuidance","pos":[520,100],"size":[317.4000244140625,58],"flags":{},"order":10,"mode":0,"inputs":[{"name":"conditioning","type":"CONDITIONING","link":175}],"outputs":[{"name":"CONDITIONING","type":"CONDITIONING","shape":3,"links":[129],"slot_index":0}],"properties":{"Node name for S&R":"FluxGuidance"},"widgets_values":[6],"color":"#233","bgcolor":"#355"},{"id":22,"type":"BasicGuider","pos":[600,0],"size":[222.3482666015625,46],"flags":{},"order":13,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":195,"slot_index":0},{"name":"conditioning","type":"CONDITIONING","link":129,"slot_index":1}],"outputs":[{"name":"GUIDER","type":"GUIDER","shape":3,"links":[30],"slot_index":0}],"properties":{"Node name for S&R":"BasicGuider"},"widgets_values":[]},{"id":67,"type":"ModelSamplingSD3","pos":[360,0],"size":[210,58],"flags":{},"order":11,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":216}],"outputs":[{"name":"MODEL","type":"MODEL","links":[195],"slot_index":0}],"properties":{"Node name for S&R":"ModelSamplingSD3"},"widgets_values":[7]},{"id":13,"type":"SamplerCustomAdvanced","pos":[860,200],"size":[272.3617858886719,124.53733825683594],"flags":{},"order":14,"mode":0,"inputs":[{"name":"noise","type":"NOISE","link":37,"slot_index":0},{"name":"guider","type":"GUIDER","link":30,"slot_index":1},{"name":"sampler","type":"SAMPLER","link":19,"slot_index":2},{"name":"sigmas","type":"SIGMAS","link":20,"slot_index":3},{"name":"latent_image","type":"LATENT","link":180,"slot_index":4}],"outputs":[{"name":"output","type":"LATENT","shape":3,"links":[181,210],"slot_index":0},{"name":"denoised_output","type":"LATENT","shape":3,"links":null}],"properties":{"Node name for S&R":"SamplerCustomAdvanced"},"widgets_values":[]},{"id":25,"type":"RandomNoise","pos":[479,618],"size":[315,82],"flags":{},"order":0,"mode":0,"inputs":[],"outputs":[{"name":"NOISE","type":"NOISE","shape":3,"links":[37]}],"properties":{"Node name for S&R":"RandomNoise"},"widgets_values":[64097275635767,"randomize"],"color":"#2a363b","bgcolor":"#3f5159"},{"id":78,"type":"TorchCompileModel","pos":[0,275.843505859375],"size":[315,58],"flags":{},"order":9,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":218}],"outputs":[{"name":"MODEL","type":"MODEL","links":[216,217],"slot_index":0}],"properties":{"Node name for S&R":"TorchCompileModel"},"widgets_values":["inductor"]},{"id":74,"type":"Note","pos":[1150,396.06298828125],"size":[210,170],"flags":{},"order":1,"mode":0,"inputs":[],"outputs":[],"properties":{},"widgets_values":["Use the tiled decode node by default because most people will need it.\n\nLower the tile_size and overlap if you run out of memory."],"color":"#432","bgcolor":"#653"},{"id":77,"type":"Note","pos":[0,0],"size":[350,110],"flags":{},"order":2,"mode":0,"inputs":[],"outputs":[],"properties":{},"widgets_values":["Select a fp8 weight_dtype if you are running out of memory."],"color":"#432","bgcolor":"#653"},{"id":11,"type":"DualCLIPLoader","pos":[0,376.3634033203125],"size":[350,106],"flags":{},"order":3,"mode":0,"inputs":[],"outputs":[{"name":"CLIP","type":"CLIP","shape":3,"links":[205],"slot_index":0}],"properties":{"Node name for S&R":"DualCLIPLoader"},"widgets_values":["clip_l.safetensors","llava_llama3_fp16.safetensors","hunyuan_video","default"]},{"id":10,"type":"VAELoader","pos":[0,521.8173828125],"size":[350,60],"flags":{},"order":4,"mode":0,"inputs":[],"outputs":[{"name":"VAE","type":"VAE","shape":3,"links":[206,211],"slot_index":0}],"properties":{"Node name for S&R":"VAELoader"},"widgets_values":["hunyuan_video_vae_bf16.safetensors"]},{"id":79,"type":"VHS_VideoCombine","pos":[1410,636.1119995117188],"size":[214.7587890625,694.2378540039062],"flags":{},"order":18,"mode":0,"inputs":[{"name":"images","type":"IMAGE","link":224},{"name":"audio","type":"AUDIO","shape":7,"link":null},{"name":"meta_batch","type":"VHS_BatchManager","shape":7,"link":null},{"name":"vae","type":"VAE","shape":7,"link":null}],"outputs":[{"name":"Filenames","type":"VHS_FILENAMES","links":null}],"properties":{"Node name for S&R":"VHS_VideoCombine"},"widgets_values":{"frame_rate":24,"loop_count":0,"filename_prefix":"HunyuanVideo","format":"video/h264-mp4","pix_fmt":"yuv420p","crf":19,"save_metadata":true,"trim_to_audio":false,"pingpong":false,"save_output":true,"videopreview":{"hidden":false,"paused":false,"params":{"filename":"HunyuanVideo_00023.mp4","subfolder":"","type":"output","format":"video/h264-mp4","frame_rate":24,"workflow":"HunyuanVideo_00023.png","fullpath":"C:\\ML\\SwarmUI\\dlbackend\\comfy\\ComfyUI\\output\\HunyuanVideo_00023.mp4"}}}},{"id":8,"type":"VAEDecode","pos":[1150,90],"size":[210,46],"flags":{},"order":15,"mode":2,"inputs":[{"name":"samples","type":"LATENT","link":181},{"name":"vae","type":"VAE","link":206}],"outputs":[{"name":"IMAGE","type":"IMAGE","links":[],"slot_index":0}],"properties":{"Node name for S&R":"VAEDecode"},"widgets_values":[]},{"id":75,"type":"SaveAnimatedWEBP","pos":[1410,200],"size":[315,366],"flags":{},"order":17,"mode":2,"inputs":[{"name":"images","type":"IMAGE","link":223}],"outputs":[],"properties":{},"widgets_values":["HunyuanVideo",24,false,80,"default"]},{"id":73,"type":"VAEDecodeTiled","pos":[1150,200],"size":[210,150],"flags":{},"order":16,"mode":0,"inputs":[{"name":"samples","type":"LATENT","link":210},{"name":"vae","type":"VAE","link":211}],"outputs":[{"name":"IMAGE","type":"IMAGE","links":[223,224],"slot_index":0}],"properties":{"Node name for S&R":"VAEDecodeTiled"},"widgets_values":[256,64,64,8]},{"id":12,"type":"UNETLoader","pos":[0,150.90908813476562],"size":[350,82],"flags":{},"order":5,"mode":0,"inputs":[],"outputs":[{"name":"MODEL","type":"MODEL","shape":3,"links":[218],"slot_index":0}],"properties":{"Node name for S&R":"UNETLoader"},"widgets_values":["Hunyuan\\hunyuan_video_t2v_720p_bf16.safetensors","fp8_e4m3fn_fast"],"color":"#223","bgcolor":"#335"},{"id":16,"type":"KSamplerSelect","pos":[484,751],"size":[315,58],"flags":{},"order":6,"mode":0,"inputs":[],"outputs":[{"name":"SAMPLER","type":"SAMPLER","shape":3,"links":[19]}],"properties":{"Node name for S&R":"KSamplerSelect"},"widgets_values":["dpmpp_2m"]},{"id":17,"type":"BasicScheduler","pos":[478,860],"size":[315,106],"flags":{},"order":12,"mode":0,"inputs":[{"name":"model","type":"MODEL","link":217,"slot_index":0}],"outputs":[{"name":"SIGMAS","type":"SIGMAS","shape":3,"links":[20]}],"properties":{"Node name for S&R":"BasicScheduler"},"widgets_values":["beta",30,1]},{"id":45,"type":"EmptyHunyuanLatentVideo","pos":[475.540771484375,432.673583984375],"size":[315,130],"flags":{},"order":7,"mode":0,"inputs":[],"outputs":[{"name":"LATENT","type":"LATENT","links":[180],"slot_index":0}],"properties":{"Node name for S&R":"EmptyHunyuanLatentVideo"},"widgets_values":[720,1280,129,1]},{"id":44,"type":"CLIPTextEncode","pos":[420,200],"size":[422.84503173828125,164.31304931640625],"flags":{},"order":8,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":205}],"outputs":[{"name":"CONDITIONING","type":"CONDITIONING","links":[175],"slot_index":0}],"title":"CLIP Text Encode (Positive Prompt)","properties":{"Node name for S&R":"CLIPTextEncode"},"widgets_values":["cinematic shot of a redhead mermaid with long hair and tail with glittering scales, swimming slowly in between submerged cliffs with coal reefs, surrounded by tropical fish, city of atlantis, fantasy, underwater scene, with beams of sunlight filtering down from the surface above, realistic style, professional high-definition video"],"color":"#232","bgcolor":"#353"}],"links":[[19,16,0,13,2,"SAMPLER"],[20,17,0,13,3,"SIGMAS"],[30,22,0,13,1,"GUIDER"],[37,25,0,13,0,"NOISE"],[129,26,0,22,1,"CONDITIONING"],[175,44,0,26,0,"CONDITIONING"],[180,45,0,13,4,"LATENT"],[181,13,0,8,0,"LATENT"],[195,67,0,22,0,"MODEL"],[205,11,0,44,0,"CLIP"],[206,10,0,8,1,"VAE"],[210,13,0,73,0,"LATENT"],[211,10,0,73,1,"VAE"],[216,78,0,67,0,"MODEL"],[217,78,0,17,0,"MODEL"],[218,12,0,78,0,"MODEL"],[223,73,0,75,0,"IMAGE"],[224,73,0,79,0,"IMAGE"]],"groups":[],"config":{},"extra":{"ds":{"scale":0.9090909090909091,"offset":[75.6822131189224,216.63708378895026]},"groupNodes":{},"VHS_latentpreview":false,"VHS_latentpreviewrate":0,"VHS_MetadataImage":true,"VHS_KeepIntermediate":true},"version":0.4}
```

### Debug Logs

```powershell
# ComfyUI Error Report
## Error Details
- **Node ID:** 13
- **Node Type:** SamplerCustomAdvanced
- **Exception Type:** ValueError
- **Exception Message:** Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
## Stack Trace

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 327, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 202, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 174, in _map_node_over_list
process_inputs(input_dict, i)

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\execution.py", line 163, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy_extras\nodes_custom_sampler.py", line 651, in sample
samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 985, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 953, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 936, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 715, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\k_diffusion\sampling.py", line 741, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 379, in __call__
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 916, in __call__
return self.predict_noise(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 919, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 359, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 195, in calc_cond_batch
return executor.execute(model, conds, x_in, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\samplers.py", line 308, in _calc_cond_batch
output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\model_base.py", line 132, in apply_model
return comfy.patcher_extension.WrapperExecutor.new_class_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\patcher_extension.py", line 110, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\model_base.py", line 163, in _apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_dynamo\eval_frame.py", line 574, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\ldm\hunyuan_video\model.py", line 316, in forward
def forward(self, x, timestep, context, y, guidance=None, attention_mask=None, control=None, transformer_options={}, **kwargs):

File "C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\comfy\ldm\hunyuan_video\model.py", line 220, in forward_orig
def forward_orig(

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_dynamo\eval_frame.py", line 745, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_functorch\aot_autograd.py", line 1184, in forward
return compiled_fn(full_args)
^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 323, in runtime_wrapper
all_outs = call_func_at_runtime_with_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_functorch\_aot_autograd\utils.py", line 126, in call_func_at_runtime_with_args
out = normalize_as_list(f(args))
^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 672, in inner_fn
outs = compiled_fn(args)
^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_functorch\_aot_autograd\runtime_wrappers.py", line 490, in wrapper
return compiled_fn(runtime_args)
^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_inductor\output_code.py", line 466, in __call__
return self.current_callable(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_inductor\utils.py", line 2128, in run
return model(new_inputs)
^^^^^^^^^^^^^^^^^

File "C:\Users\USER\AppData\Local\Temp\torchinductor_USER\w3\cw3t7d4rcaaunrnvaxb3lhpjc7mhcnps7yt4436takzlnpbufiup.py", line 748, in call
triton_poi_fused__scaled_mm__to_copy_ones_4.run(arg5_1, buf6, 3072, grid=grid(3072), stream=stream0)

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\torch\_inductor\runtime\triton_heuristics.py", line 1079, in run
return launcher(
^^^^^^^^^

File "", line 13, in launcher

File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\triton\backends\nvidia\driver.py", line 487, in __call__
self.launch(*args, **kwargs)

## System Information
- **ComfyUI Version:** 0.3.15
- **Arguments:** ComfyUI\main.py --use-sage-attention --lowvram --port 7821 --extra-model-paths-config C:\ML\SwarmUI\Data/comfy-auto-model.yaml --fast
- **OS:** nt
- **Python Version:** 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
- **Embedded Python:** true
- **PyTorch Version:** 2.6.0+cu124
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
- **Type:** cuda
- **VRAM Total:** 25756696576
- **VRAM Free:** 18567407552
- **Torch VRAM Total:** 4697620480
- **Torch VRAM Free:** 43263936

## Logs

2025-02-21T22:31:59.725106 - Adding extra search path checkpoints C:\ML\SwarmUI\Models\Stable-Diffusion
2025-02-21T22:31:59.725106 - Adding extra search path vae C:\ML\SwarmUI\Models\VAE
2025-02-21T22:31:59.725106 - Adding extra search path loras C:\ML\SwarmUI\Models\Lora
2025-02-21T22:31:59.725106 - Adding extra search path loras C:\ML\SwarmUI\Models\LyCORIS
2025-02-21T22:31:59.725106 - Adding extra search path upscale_models C:\ML\SwarmUI\Models\ESRGAN
2025-02-21T22:31:59.725106 - Adding extra search path upscale_models C:\ML\SwarmUI\Models\RealESRGAN
2025-02-21T22:31:59.725106 - Adding extra search path upscale_models C:\ML\SwarmUI\Models\SwinIR
2025-02-21T22:31:59.725106 - Adding extra search path upscale_models C:\ML\SwarmUI\Models\upscale-models
2025-02-21T22:31:59.725106 - Adding extra search path upscale_models C:\ML\SwarmUI\Models\upscale_models
2025-02-21T22:31:59.725106 - Adding extra search path embeddings C:\ML\SwarmUI\Models\Embeddings
2025-02-21T22:31:59.725106 - Adding extra search path hypernetworks C:\ML\SwarmUI\Models\hypernetworks
2025-02-21T22:31:59.725106 - Adding extra search path controlnet C:\ML\SwarmUI\Models\controlnet
2025-02-21T22:31:59.725106 - Adding extra search path clip C:\ML\SwarmUI\Models\clip
2025-02-21T22:31:59.725106 - Adding extra search path clip_vision C:\ML\SwarmUI\Models\clip_vision
2025-02-21T22:31:59.725106 - Adding extra search path unet C:\ML\SwarmUI\Models\unet
2025-02-21T22:31:59.725106 - Adding extra search path diffusion_models C:\ML\SwarmUI\Models\diffusion_models
2025-02-21T22:31:59.725106 - Adding extra search path gligen C:\ML\SwarmUI\Models\gligen
2025-02-21T22:31:59.725106 - Adding extra search path ipadapter C:\ML\SwarmUI\Models\ipadapter
2025-02-21T22:31:59.725106 - Adding extra search path yolov8 C:\ML\SwarmUI\Models\yolov8
2025-02-21T22:31:59.725106 - Adding extra search path tensorrt C:\ML\SwarmUI\Models\tensorrt
2025-02-21T22:31:59.725106 - Adding extra search path clipseg C:\ML\SwarmUI\Models\clipseg
2025-02-21T22:31:59.725106 - Adding extra search path style_models C:\ML\SwarmUI\Models\style_models
2025-02-21T22:31:59.725106 - Adding extra search path custom_nodes C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes
2025-02-21T22:31:59.725106 - Adding extra search path custom_nodes C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ExtraNodes
2025-02-21T22:32:00.847423 - Checkpoint files will always be loaded safely.
2025-02-21T22:32:00.936248 - Total VRAM 24564 MB, total RAM 126606 MB
2025-02-21T22:32:00.936248 - pytorch version: 2.6.0+cu124
2025-02-21T22:32:02.148875 - xformers version: 0.0.29.post2
2025-02-21T22:32:02.148875 - Set vram state to: LOW_VRAM
2025-02-21T22:32:02.148875 - Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
2025-02-21T22:32:02.293014 - Using sage attention
2025-02-21T22:32:03.257508 - ComfyUI version: 0.3.15
2025-02-21T22:32:03.272625 - [Prompt Server] web root: C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\web
2025-02-21T22:32:03.690425 -
Import times for custom nodes:
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\custom_nodes\websocket_image_save.py
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes\ComfyUI-APG_ImYourCFGNow
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes\sd-dynamic-thresholding
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes\ComfyUI-SAI_API
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes\Skimmed_CFG
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ExtraNodes\SwarmComfyExtra
2025-02-21T22:32:03.690425 - 0.0 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\DLNodes\ComfyUI-GGUF
2025-02-21T22:32:03.690425 - 0.1 seconds: C:\ML\SwarmUI\src\BuiltinExtensions\ComfyUIBackend\ExtraNodes\SwarmComfyCommon
2025-02-21T22:32:03.690425 - 0.1 seconds: C:\ML\SwarmUI\dlbackend\comfy\ComfyUI\custom_nodes\ComfyUI-VideoHelperSuite
2025-02-21T22:32:03.690425 -
2025-02-21T22:32:03.694428 - Starting server

2025-02-21T22:32:03.694428 - To see the GUI go to: http://127.0.0.1:7821
2025-02-21T22:34:09.670123 - got prompt
2025-02-21T22:34:09.708648 - Using xformers attention in VAE
2025-02-21T22:34:09.709654 - Using xformers attention in VAE
2025-02-21T22:34:09.961512 - VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
2025-02-21T22:34:10.154511 - model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
2025-02-21T22:34:10.155511 - model_type FLOW
2025-02-21T22:34:22.227459 - Requested to load HunyuanVideoClipModel_
2025-02-21T22:34:22.233458 - loaded completely 9.5367431640625e+25 14550.85205078125 True
2025-02-21T22:34:22.235462 - CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16
2025-02-21T22:34:22.284459 - clip missing: ['text_projection.weight']
2025-02-21T22:34:35.426817 - Requested to load HunyuanVideo
2025-02-21T22:34:36.816200 - loaded partially 5286.799999999999 5286.494201660156 0
2025-02-21T22:51:35.717738 -
100%|##########| 30/30 [16:58<00:00, 32.76s/it]2025-02-21T22:51:35.717738 -
100%|##########| 30/30 [16:58<00:00, 33.96s/it]2025-02-21T22:51:35.717738 -
2025-02-21T22:51:35.755944 - Requested to load AutoencoderKL
2025-02-21T22:51:37.122599 - 0 models unloaded.
2025-02-21T22:51:37.137711 - loaded partially 64.0 63.99985313415527 0
2025-02-21T22:52:23.001270 - Prompt executed in 1093.33 seconds
2025-02-21T22:54:37.700011 - got prompt
2025-02-21T22:54:37.872111 - Requested to load HunyuanVideo
2025-02-21T22:54:39.038962 - loaded partially 4430.674999999999 4430.615295410156 0
2025-02-21T22:54:39.048962 -
0%| | 0/30 [00:00", line 13, in launcher
File "C:\ML\SwarmUI\dlbackend\comfy\python_embeded\Lib\site-packages\triton\backends\nvidia\driver.py", line 487, in __call__
self.launch(*args, **kwargs)
ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)

2025-02-21T22:54:39.083962 - Prompt executed in 1.38 seconds

Note that the first run of the workflow executes successfully, and the second one immediately fails.
```

### Other

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.