[ROCm][gfx1201] General model loading became extremely slow on R9700 - LTX 2.3 workflow increased from about 300s to about 4400s
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Your question
**Description**
This does not appear to be a DualCLIPLoader-only issue.
The slowdown affects model loading more generally across the entire workflow.
Until approximately July 18-19, 2026, my LTX Director 2 / LTX 2.3 workflow consistently worked at normal speed.
DualCLIPLoader loaded both the Gemma and LTX text encoders almost immediately, usually within a few seconds.
The models used by DualCLIPLoader are:
- "gemma_3_12B_it_fp4_mixed.safetensors"
- "ltx-2.3_text_projection_bf16.safetensors"
SamplerCustomAdvanced also loaded the LTX 2.3 KModel into VRAM almost immediately. VRAM usage increased very quickly and sampling started without a noticeable delay.
A 15-second video at 24 fps and approximately 648p, including the first model loading and the first render, normally completed in around 300 seconds.
Now the exact same workflow, models and settings take around 4400 seconds, approximately 1 hour and 14 minutes.
This is not a first-run versus second-run slowdown.
The workflow worked consistently and repeatedly at normal speed for an extended period. Around July 18-19, something changed, and since then every run has remained extremely slow, including the first run after a complete system reboot.
**Actual behavior**
The slowdown affects multiple stages of model loading.
1. DualCLIPLoader
DualCLIPLoader now takes several minutes, and sometimes 10-15 minutes, to load approximately 5-8.5 GB of encoder data.
The log remains for a very long time at:
```text
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
```
It eventually continues, so ComfyUI is not completely frozen.
Previously, both encoders loaded within a few seconds.
The affected encoder files are:
```text
gemma_3_12B_it_fp4_mixed.safetensors
ltx-2.3_text_projection_bf16.safetensors
```
DualCLIPLoader settings:
```text
type: ltxv
device: default
```
**2. SamplerCustomAdvanced / KModel loading**
The same slowdown occurs later when SamplerCustomAdvanced needs the LTX 2.3 KModel.
Previously, the KModel loaded into VRAM almost immediately. VRAM usage jumped very quickly and sampling started.
Now VRAM fills very slowly, little by little.
Loading the KModel can take several minutes before sampling even begins.
The slowdown therefore does not seem to be limited to DualCLIPLoader, Gemma or text encoder initialization. It appears to affect general model loading and transfers to VRAM across the workflow.
#### 3. Overall generation performance
The slowdown is not limited to the initial loading stage.
Previously:
```text
Complete generation time: approximately 300 seconds
Video duration: 15 seconds
Frame rate: 24 fps
Resolution: approximately 648p
```
Currently:
```text
Complete generation time: approximately 4400 seconds
Equivalent time: approximately 1 hour and 14 minutes
```
The workflow, models, duration, frame rate and resolution are unchanged.
**Expected behavior**
- DualCLIPLoader should load the Gemma and LTX encoders within a few seconds.
- The LTX 2.3 KModel should load into VRAM almost immediately.
- VRAM usage should increase quickly while models are loaded.
- SamplerCustomAdvanced should start sampling without a delay of several minutes.
- The complete workflow should take approximately 300 seconds, as it consistently did before July 18-19.
### Steps to reproduce
1. Start ComfyUI after a complete system reboot.
2. Load the attached LTX Director 2 / LTX 2.3 workflow.
3. Use DualCLIPLoader with:
```text
clip_name1: gemma_3_12B_it_fp4_mixed.safetensors
clip_name2: ltx-2.3_text_projection_bf16.safetensors
type: ltxv
device: default
```
4. Use the same LTX 2.3 KModel selected in the attached workflow.
5. Queue a 15-second video at 24 fps and approximately 648p.
6. Observe the long delay after:
```text
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
```
7. Observe VRAM usage while the KModel is loaded before SamplerCustomAdvanced begins sampling.
8. VRAM fills slowly and progressively instead of increasing almost immediately.
9. The complete workflow takes approximately 4400 seconds instead of approximately 300 seconds.
**Environment**
- GPU: AMD Radeon AI PRO R9700 32 GB
- GPU architecture: native `gfx1201`
- GPU VRAM detected by ComfyUI: 32624 MB
- System RAM: 64 GB
- OS: Ubuntu 24.04.4 LTS
- Kernel: `7.0.0-28-generic`
- Python: `3.12.3`
- PyTorch: `2.12.0+rocm7.14.0`
- ROCm/HIP runtime: `7.14.60850`
- ROCm version reported by ComfyUI: `7.14`
- Triton: `3.7.1+git0263a6a6.rocm7.14.0`
- ComfyUI: `0.28.2`
- ComfyUI channel: latest master
- ComfyUI frontend: `1.45.21`
- ComfyUI Manager: `4.2.2`
- comfy-kitchen: `0.2.22`
- comfy-aimdo: `0.4.10`
- Attention backend: PyTorch attention
- VRAM mode: NORMAL_VRAM
- Workflow: LTX Director 2 / LTX 2.3
- Video duration: approximately 15 seconds
- Frame rate: 24 fps
- Output resolution: approximately 648p
**Text encoders**
```text
gemma_3_12B_it_fp4_mixed.safetensors
ltx-2.3_text_projection_bf16.safetensors
```
**LTX components used by the workflow**
```text
LTX Director 2 / LTX 2.3 KModel
LTX23_video_vae_bf16.safetensors
LTX23_audio_vae_bf16.safetensors
taeltx2_3.safetensors
ltx-2.3-spatial-upscaler-x2-1.1.safetensors
```
The exact KModel filename and all model selections are also included in the attached workflow JSON.
**Custom** nodes
The workflow uses current versions of:
- ComfyUI-LTXVideo
- ComfyUI-KJNodes
- ComfyUI-GGUF
- WhatDreamsCost-ComfyUI
- ComfyUI-AMDGPUMonitor
- ComfyUI Manager
The slowdown is visible in core workflow stages involving:
- DualCLIPLoader
- SamplerCustomAdvanced
- KModel loading
- transfers of models into VRAM
**GPU detection**
The GPU is detected correctly and runs natively as `gfx1201`:
```text
Total VRAM 32624 MB, total RAM 63602 MB
pytorch version: 2.12.0+rocm7.14.0
AMD arch: gfx1201
ROCm version: (7, 14)
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon AI PRO R9700 : native
Using pytorch attention
```
A separate PyTorch GPU matrix multiplication test completes successfully:
```text
GPU available: True
GPU count: 1
GPU: AMD Radeon AI PRO R9700
Architecture includes: gfx1201
GPU matrix multiplication test: OK
```
### Troubleshooting already performed
I have already:
- restored the system using Timeshift
- checked the system for package or filesystem problems after the Timeshift restore
- reinstalled Ubuntu
- tested different recent ComfyUI versions
- updated ComfyUI to the latest master
- reinstalled the PyTorch and ROCm environment
- updated ComfyUI Manager
- updated all custom nodes
- reinstalled ComfyUI-GGUF
- reinstalled ComfyUI-KJNodes
- installed the current ComfyUI-LTXVideo nodes
- verified native `gfx1201` detection
- removed the old `HSA_OVERRIDE_GFX_VERSION=11.0.0` override
- removed the invalid `ROCM_PATH=/opt/rocm` setting
- confirmed access to `/dev/kfd` and `/dev/dri/renderD128`
- added the user to the `render` and `video` groups
- successfully performed a PyTorch GPU matrix multiplication test
- tested after complete system reboots
- tested with async weight offloading disabled
- tested with pinned memory disabled
- tested with dynamic VRAM disabled
- checked available system RAM
- checked available VRAM
- checked swap usage
- checked kernel logs for AMDGPU errors
- checked Python dependencies using `pip check`
None of these tests restored the original loading or generation speed.
**Errors not present**
There are no:
- GPU resets
- GPU page faults
- AMDGPU VM faults
- OOM errors
- ring timeouts
- KFD errors
- kernel crashes
- broken Python package requirements
- missing GPU devices
- incorrect GPU architecture detection
System memory is also not exhausted.
During testing:
```text
Available system RAM: approximately 53 GB
Swap usage: 0 GB
```
The kernel log shows the GPU and KFD initializing correctly.
**Important observation**
This does not look like only a slow text encoder or DualCLIPLoader problem.
Both text encoder loading and KModel loading are now much slower than before.
Previously, VRAM usage increased almost immediately when a model was loaded.
Now VRAM fills gradually and very slowly, as though model transfers to VRAM, model initialization or memory management have become heavily throttled.
The total generation time increased from approximately 300 seconds to approximately 4400 seconds.
This appears to be a more general regression involving one or more of the following:
- model loading
- transfers from RAM to VRAM
- ROCm memory transfers
- ComfyUI memory management
- MixedPrecisionOps initialization
- quantized model loading
- safetensors loading
- KModel loading
- comfy-kitchen
- recent ComfyUI changes
- recent PyTorch or ROCm changes
The important point is that this system and workflow worked consistently at normal speed before approximately July 18-19.
The slowdown did not gradually develop during a single session.
Something changed around that date, and since then the same workflow has remained permanently slow, including immediately after complete system reboots and system reinstalls.
[LTX_Director_2_Workflow_Hotfix.json](https://github.com/user-attachments/files/30182069/LTX_Director_2_Workflow_Hotfix.json)
### Logs
```powershell
```
### Other
_No response_
Contributor guide
Research direction
Reproduce the slowdown with the attached LTX_Director_2_Workflow_Hotfix.json using the reported DualCLIPLoader and SamplerCustomAdvanced settings. Measure encoder loading, KModel transfer, and total generation time, then compare those stages with the reported former timings. Done means the regression is isolated to a specific subsystem and a verified fix restores expected loading and generation performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100