linkedin / linkedin/Liger-Kernel

[ROCm]: State of Liger Kernel CI on AMD on ROCm 6.3

Open
#624 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
6.6k
Forks
603
Avg merge
1d 20h
Merged PRs (30d)
47

Description

### 🐛 Describe the bug

# State of Liger Kernel CI on MI300X on ROCm 6.3
## Environment:
* `python3 -m pip list | grep triton`
```
pytorch-triton-rocm 3.3.0+git96316ce5
triton 3.2.0
```

* `python3 -m pip list | grep torch`
```
pytorch-triton-rocm 3.3.0+git96316ce5
torch 2.8.0.dev20250321+rocm6.3
```

1. `make test`
This is the failure case:
`test/transformers/test_jsd.py::test_correctness_with_beta[0.9-dtype1-1e-08-1e-06-2-1024-3200]`
it seems that the `atol` of this test on AMD if relaxed to `1e-07`, the tests will pass.

2. `make test-convergence`
`test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype3-1e-08-1e-05-0.005-1e-05-0.005-1e-05]`
it seems that the `loss_atol` of this test on AMD if relaxed from `1e-8` to `1e-05`, the tests will pass.

# Follow up

I would like to get some opinions on:
1. move the AMD CI unit tests to rocm 6.3 only as it is getting more optimization and bug fixes. On the pytorch main webpage, they have also moved to rocm 6.3 (https://pytorch.org/get-started/locally/)
2. relax the tolerance for those test cases if they are still at acceptable value.

### Reproduce

_No response_

### Versions

```
Environment Report:
-------------------
Operating System: Linux-5.15.0-116-generic-x86_64-with-glibc2.35
Python version: 3.10.12
Liger Kernel version: 0.5.5
PyTorch version: 2.8.0.dev20250321+rocm6.3
CUDA version: None
HIP(ROCm) version: 6.3.42131-fa1d09cbd
Triton version: 3.3.0
Transformers version: 4.49.0
XPU version: XPU Not Available
```

Contributor guide

Open the contributing guide

Research direction

Start by running make test and make test-convergence in the reported ROCm 6.3 environment, focusing on test/transformers/test_jsd.py::test_correctness_with_beta and test/convergence/fp32/test_mini_models_with_logits.py::test_mini_model. Review the existing tolerance expectations and CI setup; done means a documented decision on ROCm coverage and any acceptable tolerance changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.