ByteDance-Seed / ByteDance-Seed/Triton-distributed

Discussion: Enabling NVIDIA-Equivalent Distributed Optimizations on AMD GPUs

Open
#108 5 comments 0 reactions 1 assignee Claimed by @wenlei-bao View on GitHub
AMD
Dominant language
Python
Stars
1.5k
Forks
172
PR merge metrics
No merged PRs in 30d

Description

Hi all,
We've noticed the powerful intra-node and inter-node optimizations in triton-distributed for NVIDIA GPUs and are very impressed. I'd like to start a discussion on **whether extending these excellent optimizations to the AMD GPU platform, especially for low-latency All-to-All, would require additional support from AMD's underlying libraries (e.g., bitcode).**
As mentioned in the triton-distributed paper, the nvshmem bitcode library's support for IBRC but not IBGDA restricts Triton to the IBRC mode, which in turn impacts performance at scales beyond 64 GPUs.
This naturally leads to our key questions regarding AMD platform support:
Underlying Library Feature Parity: **Does AMD's rocshmem (or its equivalent bitcode library) already possess all the necessary features required for implementing both intra-node and inter-node communication optimizations?**
Roadmap and Contribution: Are there any existing plans or a roadmap from the community or development team for this? If we are interested in contributing to this effort, could you provide some guidance or suggestions?
Thank you for your time and guidance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.