Pipeline parallelism
Open
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
I have a 2 node Strix halo 128GB setup (256GB total) running Deepseek V4 flash. I noticed that at all times one of the nodes is doing nothing waiting for the other node to complete its work. All the while requests are being piled up waiting to be processed.
Maybe we can do something so that multiple nodes can process different requests in parallel so they don't sit there doing nothing?
Contributor guide
Assessment
This issue has not been assessed yet.