JuliaCI / JuliaCI/BenchmarkTools.jl

Feature Idea: Custom Benchmarking Metric

Open
#176 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

Hi,

in my research we're developing an adaptive solver for some agent based simulations and we build a
benchmark suite with PkgBenchmark.
This already helps a lot but sometimes the results can be quite confusing because the runtime of the full solving process
depends very strongly on the adaptive solver and its heuristics.
One thing that would improve our benchmarks significantly would be if we could include
a custom additional metric to our benchmark pipeline.
(In this case that would just be the number of iterations)

We already have one working but inefficient way of doing this

@benchmarkable sleep(iterations/1000) setup=(iterations=solve(...))

which adds an entry in the PkgBenchmark judge/result files hinting at the number of iterations but of course this is very inefficient.

I also tried implementing a macro for this myself by essentially duplicating the code for @benmarkable but
so far this lacks generality and how this should fit into the rest of the logic is not clear.
( https://github.com/JonasIsensee/BenchmarkTools.jl/tree/mytrial )

What are your thoughts?
Would this be useful to others as well?
Could this be done in a slightly more general way?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing @benchmarkable macro and the author's trial implementation linked in the issue. Trace how benchmark results are passed into the PkgBenchmark pipeline, then determine what a general custom-metric interface would need to produce a recorded iteration count without the sleep workaround. Done means the metric works through the normal benchmarking flow and is useful beyond this specific solver case.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.