ByteDance-Seed / ByteDance-Seed/Triton-distributed

Why is there no performance improvements on H20 GPUs?

Open
#48 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
1.5k
Forks
172
PR merge metrics
No merged PRs in 30d

Description

I test Gemm + ReduceScatter On Nvdia H20 Gpus, Intra-GEMM, 1 Node 8 H20Gpus.
Trition-Distributed is slower than Torch + NCCL.
I have a question : Does Trition-Distributed need a high-performance machine such as H100、H800 or MI300X to get performance improvements?

| MNK | TritonDistributed(ms) | Torch + NCCL(ms) |
| --- | ---| ---|
8192x8192x29568 | 4.17 | 4.01
32x8192x29568 | 0.55 | 0.07

Contributor guide

Open the contributing guide

Research direction

No files or tests are identified in the issue. Start by reproducing the one-node, eight-H20 Gemm plus ReduceScatter measurements for the two listed MNK cases, then compare Triton-Distributed with Torch plus NCCL. Done means establishing whether the gap is expected on H20 hardware and documenting the relevant performance requirements or cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.