linkedin / linkedin/Liger-Kernel
Automating benchmark testing with every pr merged.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
Hello,
So we discussed an approach to solve this:
Running all the benchmarks take less than an hour. I tried it on a single H100 GPU and it took me 55 minutes.
So, what we thought that we can run the benchmark scripts after every pr is merged to the main repo using the CI.
Similar to what we do for docs, we will run a the benchmark script after anything is pushed to the main branch of the repo.
Initially the data of the benchmark will be stored in the gh-pages branch. Once the script is run (make run-benchmarks) the new data will be appended to the csv file in the gh-pages branch with another column which would be github-commit-hash. So now, for every commit we will have the benchmark data, and it will be visualized using github pages on for example: https://linkedin.github.io/Liger-Kernel/benchmarks. Using javascript the data will be visualized using charts just like (https://hud.pytorch.org/benchmark/compilers). There would be filters and user can see benchmark improvements for speed and memory.
Let me know what you guys think about this. @Tcc0403 @yundai424 @lancerts
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.