Implement Naive Model Parallel
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
Explained here: https://huggingface.co/docs/transformers/v4.13.0/parallelism#naive-model-parallel-vertical-and-pipeline-parallel
Basically, it splits model layer-by-layer across multiple GPU, and hidden states goes sequentially from one GPU to another. Only one GPU is active at the moment.
**Why:**
It's for those who has DDR4 and/or not very fast PCIe connection. Personally, I tried diffusers which has this functionality out of the box and it improved speed of my generations 2x-6x (depending on model) compared to block swapping (in either dynamic vram mode or old one).
Other parallelism solutions don't work because they require fast PCIe connection. Naive model parallel **doesn't** require it because hidden_states are pretty lightweight.
**How:**
I'd prefer to pass manual `device_map` (argument in diffusers/transformers - it's a dict where key is layer prefix and value is device. In llama.cpp key is layer regex: probably it would be more convenient that way) to Load Model node. It would allow to have full manual control of how much of model each device will get. Personally I don't need auto mode (for auto balanced ditribution across GPU), but if you implement it, it would be nice.
**Pipeline parallelism:**
Would be nice to get, but only as a next step because I'm afraid it can be too troublesome to implement right away.
### Existing Solutions
All distributed solutions described here
https://github.com/komikndr/raylight#raylight-vs-multigpu-vs-comfyui-worksplit-branch-vs-comfyui-distributed
None of them providing model or pipeline parallelism.
### Other
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.