ByteDance-Seed / ByteDance-Seed/Triton-distributed
Discussion: Enabling NVIDIA-Equivalent Distributed Optimizations on AMD GPUs
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
We've noticed the powerful intra-node and inter-node optimizations in triton-distributed for NVIDIA GPUs and are very impressed. I'd like to start a discussion on **whether extending these excellent optimizations to the AMD GPU platform, especially for low-latency All-to-All, would require additional support from AMD's underlying libraries (e.g., bitcode).**
As mentioned in the triton-distributed paper, the nvshmem bitcode library's support for IBRC but not IBGDA restricts Triton to the IBRC mode, which in turn impacts performance at scales beyond 64 GPUs.
This naturally leads to our key questions regarding AMD platform support:
Underlying Library Feature Parity: **Does AMD's rocshmem (or its equivalent bitcode library) already possess all the necessary features required for implementing both intra-node and inter-node communication optimizations?**
Roadmap and Contribution: Are there any existing plans or a roadmap from the community or development team for this? If we are interested in contributing to this effort, could you provide some guidance or suggestions?
Thank you for your time and guidance.
Contributor guide
Assessment
This issue has not been assessed yet.