AMD SVD Performance / Stability

I'm having trouble running the stable video diffusion examples on my machine.

OS: Arch linux
CPU: AMD Ryzen 9 7950X
RAM: 64GB
GPU: AMD Radeon RX 7900 XTX
VRAM: 24GB
Software: ComfyUI 329c57199302f6b9ccfebb86c96e937c386da92f, Rocm 5.6... Wait. See follow-up at the end.

When I tried running the [14 frame example](https://comfyanonymous.github.io/ComfyUI_examples/video/), it was very slow and my GPU eventually locked up. `dmesg` shows this:

```
Dec 15 17:47:10 hawk kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=20268, emitted seq=20269
Dec 15 17:47:10 hawk kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process gnome-shell pid 1718 thread gnome-shel:cs0 pid 1755
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: recover vram bo from shadow start
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: recover vram bo from shadow done
Dec 15 17:47:10 hawk kernel: [drm] ring gfx_32779.1.1 was added
Dec 15 17:47:10 hawk kernel: [drm] ring compute_32779.2.2 was added
Dec 15 17:47:10 hawk kernel: [drm] ring sdma_32779.3.3 was added
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:40 vmid:8 pasid:32779, for process pid 0 thread pid 0)
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: in page starting at address 0x0000800100269000 from client 10
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00840C50
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: Faulty UTCL2 client ID: CPG (0x6)
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: MORE_FAULTS: 0x0
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: WALKER_ERROR: 0x0
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: PERMISSION_FAULTS: 0x5
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: MAPPING_ERROR: 0x0
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: RW: 0x1
Dec 15 17:47:10 hawk kernel: [drm] ring gfx_32779.1.1 ib test pass
Dec 15 17:47:10 hawk kernel: [drm] ring compute_32779.2.2 ib test pass
Dec 15 17:47:10 hawk kernel: [drm] ring sdma_32779.3.3 ib test pass
Dec 15 17:47:10 hawk kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset(2) succeeded!
Dec 15 17:48:00 hawk gnome-shell[1718]: amdgpu: The CS has been rejected (-125), but the context isn't robust.
Dec 15 17:48:00 hawk gnome-shell[1718]: amdgpu: The process will be terminated.
Dec 15 17:48:00 hawk kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
```

That was after I added the `iommu=soft` kernel parameter. Before I would see `IO_PAGE_FAULT` in the logs, among other things. I'm not sure if it's particularly interesting to see the details.

The previous GPU crash (before setting `iommu=soft`) started with this:
```
Dec 14 21:40:41 hawk kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=1788838, emitted seq=1788840
Dec 14 21:40:42 hawk kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process kgx pid 2013 thread kgx:cs0 pid 2414
```

I'm also getting ~60s/it, which seems terribly slow. In regular sd-1.5 I get 10-15it/s for smaller (512x512) images, and it works great generating image after image.

If I run SVD with reduced settings, I can get through the process and produce a video, but it's still very slow.

I tried all the different cross-attention methods, tried forcing fp16 and fp32, tried highvram and disable-smart-memory. Nothing changed the speed appreciably.

Any idea what might be going on here?

[Update] Ok, when I went to write down what versions of software I was running, I noticed I had ROCm 5.6. I installed 5.7, and now I get 3-4s/it in KSampler, and the whole prompt finished in 181s. I think this is solved, but I'll submit the issue anyway, if only for the record.