JuliaGaussianProcesses / JuliaGaussianProcesses/KernelFunctions.jl

Benchmarking!

Open
#387 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
275
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Related to #386, I would like to open the discussion for evaluating the performance of basic functions.
Here are some facts/ideas.

Existing tools

  • PkgBenchmark.jl: A nice tool to create a suite of benchmarks and has functions to create nice reports on variations of performance. You can even create a markdown report which could be posted on the PR
  • NanoSoldier.jl: The tool used by Julia to evaluate the performance of the language. I don't believe this can be adapted easily to our setup, but I did not checked the details
  • Github action benchmark: Given a benchmark output will also create a report and has the possibility to directly create comments

Potential issues

  • Benchmark is highly dependent on the machine used, if we use the Github clusters, we might get a large variance in the results depending on the time of the day etc...
  • We cannot benchmark everything, which means we need to restrict ourselves to maybe the most used functions/kernels etc
  • Adding benchmarks can be a lot of work, can we find a framework where adding new tests is smooth

Other ideas

  • Not all PRs are performance-critical, we should be able to call whatever tool we use only when needed? Maybe every time for master and at will for some PRs.
  • What do we want to benchmark? Only pairwise, kernelmatrix or also the performance of the gradients on them?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read related issue #386 and compare the listed options: PkgBenchmark.jl, NanoSoldier.jl, and GitHub Action benchmark. Clarify which functions or gradients to measure, how benchmarks should be triggered, and how machine variance will be handled. This is done when the project has an agreed benchmarking scope and framework.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, julia
Domain
ci-cd, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.