ByteDance-Seed / ByteDance-Seed/Triton-distributed

[Bug] Run into CUDA error: misaligned address on H100

Open
#22 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.5k
Forks
172
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

`./third_party/distributed/launch.sh ./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py --case correctness_tma`

Results in bug:
```
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/aiscuser/cy/Triton-distributed/./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py", line 275, in
[rank0]: func(args)
[rank0]: File "/home/aiscuser/cy/Triton-distributed/./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py", line 120, in test_ag_gemm_tma_intra_node
[rank0]: torch.cuda.synchronize()
[rank0]: File "/home/aiscuser/miniconda3/envs/tilelink/lib/python3.9/site-packages/torch/cuda/__init__.py", line 892, in synchronize
[rank0]: return torch._C._cuda_synchronize()
[rank0]: RuntimeError: CUDA error: misaligned address
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```
It seems that the bug is caused by TMA? Could you please help me take a look, thanks.

### Environment details

GPU: NVIDIA H100 80GB HBM3 * 4
GPU topo:
GPU0 GPU1 GPU2 GPU3
GPU0 X NV18 NV18 NV18
GPU1 NV18 X NV18 NV18
GPU2 NV18 NV18 X NV18
GPU3 NV18 NV18 NV18 X

CUDA Version: 12.2
Driver Version: 535.86.10

Contributor guide

Open the contributing guide

Research direction

Start with third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py, especially test_ag_gemm_tma_intra_node, and run the reported launch.sh command with CUDA_LAUNCH_BLOCKING=1. Trace the correctness_tma path to determine whether TMA causes the misaligned address and establish a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.