JuliaGaussianProcesses / JuliaGaussianProcesses/KernelFunctions.jl
Benchmarking!
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 275
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
Related to #386, I would like to open the discussion for evaluating the performance of basic functions.
Here are some facts/ideas.
Existing tools
- PkgBenchmark.jl: A nice tool to create a suite of benchmarks and has functions to create nice reports on variations of performance. You can even create a markdown report which could be posted on the PR
- NanoSoldier.jl: The tool used by Julia to evaluate the performance of the language. I don't believe this can be adapted easily to our setup, but I did not checked the details
- Github action benchmark: Given a benchmark output will also create a report and has the possibility to directly create comments
Potential issues
- Benchmark is highly dependent on the machine used, if we use the Github clusters, we might get a large variance in the results depending on the time of the day etc...
- We cannot benchmark everything, which means we need to restrict ourselves to maybe the most used functions/kernels etc
- Adding benchmarks can be a lot of work, can we find a framework where adding new tests is smooth
Other ideas
- Not all PRs are performance-critical, we should be able to call whatever tool we use only when needed? Maybe every time for master and at will for some PRs.
- What do we want to benchmark? Only
pairwise,kernelmatrixor also the performance of the gradients on them?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read related issue #386 and compare the listed options: PkgBenchmark.jl, NanoSoldier.jl, and GitHub Action benchmark. Clarify which functions or gradients to measure, how benchmarks should be triggered, and how machine variance will be handled. This is done when the project has an agreed benchmarking scope and framework.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, julia
- Domain
- ci-cd, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100