ROCm / ROCm/FlyDSL

[Feature]: Move benchmark/perf test from test folder to FlyDSL python source

Open
#304 1 comment 0 reactions 2 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
282
Forks
120
Avg merge
1d 21h
Merged PRs (30d)
67

Description

Suggestion Description

I would suggest moving common benchmarking/perf test(https://github.com/ROCm/FlyDSL/blob/main/tests/test_common.py) from test folder to FlyDSL python source, so it can be imported directly like this

An example,
from fx.testing import checkAllclose, run_perftest

Why should this be done

  1. Triton DSL and even CuteDSL provides such feature as part of their API. See (https://triton-lang.org/main/python-api/triton.testing.html and https://github.com/triton-lang/triton/blob/main/python/triton/testing.py). CuteDSL also provides a similar API. See this example code https://github.com/NVIDIA/cutlass/blob/main/examples/python/CuTeDSL/notebooks/elementwise_add.ipynb
  2. We almost always have benchmark code for most of the kernels. For example, Triton kernels in AITER have a whole suite of benchmarks for kernels. https://github.com/ROCm/aiter/tree/main/op_tests/op_benchmarks/triton. They all use the common benchmarking code provided by Triton DSL. Some examples.
    https://github.com/ROCm/aiter/blob/main/op_tests/op_benchmarks/triton/bench_gemm_afp4wfp4.py#L51
    https://github.com/ROCm/aiter/blob/main/op_tests/op_benchmarks/triton/bench_batch_prefill.py#L158

As we develop more FlyDSL kernels, I expect similar benchmarking suite to exist for them. Using common benchmarking code/API that is provided by FlyDSL would be useful here.

Operating System

No response

GPU

No response

ROCm Component

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.