ByteDance-Seed / ByteDance-Seed/Triton-distributed
[Bug] Run into CUDA error: misaligned address on H100
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
`./third_party/distributed/launch.sh ./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py --case correctness_tma`
Results in bug:
```
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/aiscuser/cy/Triton-distributed/./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py", line 275, in
[rank0]: func(args)
[rank0]: File "/home/aiscuser/cy/Triton-distributed/./third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py", line 120, in test_ag_gemm_tma_intra_node
[rank0]: torch.cuda.synchronize()
[rank0]: File "/home/aiscuser/miniconda3/envs/tilelink/lib/python3.9/site-packages/torch/cuda/__init__.py", line 892, in synchronize
[rank0]: return torch._C._cuda_synchronize()
[rank0]: RuntimeError: CUDA error: misaligned address
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```
It seems that the bug is caused by TMA? Could you please help me take a look, thanks.
### Environment details
GPU: NVIDIA H100 80GB HBM3 * 4
GPU topo:
GPU0 GPU1 GPU2 GPU3
GPU0 X NV18 NV18 NV18
GPU1 NV18 X NV18 NV18
GPU2 NV18 NV18 X NV18
GPU3 NV18 NV18 NV18 X
CUDA Version: 12.2
Driver Version: 535.86.10
Contributor guide
Research direction
Start with third_party/distributed/distributed/test/nvidia/test_ag_gemm_intra_node.py, especially test_ag_gemm_tma_intra_node, and run the reported launch.sh command with CUDA_LAUNCH_BLOCKING=1. Trace the correctness_tma path to determine whether TMA causes the misaligned address and establish a reproducible resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100