Multi-GPU Support

Enhancement Request/Not Issue

Windows 10 (64GB RAM, 4090, 3080, i7 4930k)

Been working on this for a few days, without success.

Attempting to leverage both of my GPU's to decrease iteration time on a single job.

Went the accelerate route, worked with multiple CUDA variants, built from source multiple times with torch. Can get both gpu's seen no problem, then the libuv not found issue begins. Work-arounds to get gloo seen has been an absolute nightmare. Initialized torch.distributed.launch before everything else starts loading, and multitudes of issues start arising. One route I run into CUDA OOM, another for offloading to CPU it never works even though 256GB NVME virtual memory is available due to various issues. Specifying different models/vae to offload to different RANKS/GPU's seems to be the way to go but stuck on _no_split_module for the models and then fails.

Tried DeepSpeed route with parallism. Wondering if anyone has got multi-gpu support down yet.

Closest I have got was when I get the error (no_split_modules) for hyunan models.

Anyone cross the finish line on this one?

Fantastic work, otherwise putting this together, IIIyasviel!

No repository files or tests are identified in the issue. Begin by locating the existing model-loading and GPU initialization paths, then compare them with the reported accelerate, torch.distributed.launch, and DeepSpeed attempts. The issue does not define a concrete completion criterion beyond supporting multiple GPUs for one job.