Comfy-Org / Comfy-Org/ComfyUI

Implement Naive Model Parallel

Open
#13,537 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

Explained here: https://huggingface.co/docs/transformers/v4.13.0/parallelism#naive-model-parallel-vertical-and-pipeline-parallel

Basically, it splits model layer-by-layer across multiple GPU, and hidden states goes sequentially from one GPU to another. Only one GPU is active at the moment.

**Why:**
It's for those who has DDR4 and/or not very fast PCIe connection. Personally, I tried diffusers which has this functionality out of the box and it improved speed of my generations 2x-6x (depending on model) compared to block swapping (in either dynamic vram mode or old one).
Other parallelism solutions don't work because they require fast PCIe connection. Naive model parallel **doesn't** require it because hidden_states are pretty lightweight.

**How:**
I'd prefer to pass manual `device_map` (argument in diffusers/transformers - it's a dict where key is layer prefix and value is device. In llama.cpp key is layer regex: probably it would be more convenient that way) to Load Model node. It would allow to have full manual control of how much of model each device will get. Personally I don't need auto mode (for auto balanced ditribution across GPU), but if you implement it, it would be nice.

**Pipeline parallelism:**
Would be nice to get, but only as a next step because I'm afraid it can be too troublesome to implement right away.

### Existing Solutions

All distributed solutions described here
https://github.com/komikndr/raylight#raylight-vs-multigpu-vs-comfyui-worksplit-branch-vs-comfyui-distributed

None of them providing model or pipeline parallelism.

### Other

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.