JuliaCI / JuliaCI/BenchmarkTools.jl
Feature Idea: Custom Benchmarking Metric
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 684
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
in my research we're developing an adaptive solver for some agent based simulations and we build a
benchmark suite with PkgBenchmark.
This already helps a lot but sometimes the results can be quite confusing because the runtime of the full solving process
depends very strongly on the adaptive solver and its heuristics.
One thing that would improve our benchmarks significantly would be if we could include
a custom additional metric to our benchmark pipeline.
(In this case that would just be the number of iterations)
We already have one working but inefficient way of doing this
@benchmarkable sleep(iterations/1000) setup=(iterations=solve(...))
which adds an entry in the PkgBenchmark judge/result files hinting at the number of iterations but of course this is very inefficient.
I also tried implementing a macro for this myself by essentially duplicating the code for @benmarkable but
so far this lacks generality and how this should fit into the rest of the logic is not clear.
( https://github.com/JonasIsensee/BenchmarkTools.jl/tree/mytrial )
What are your thoughts?
Would this be useful to others as well?
Could this be done in a slightly more general way?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing @benchmarkable macro and the author's trial implementation linked in the issue. Trace how benchmark results are passed into the PkgBenchmark pipeline, then determine what a general custom-metric interface would need to produce a recorded iteration count without the sleep workaround. Done means the metric works through the normal benchmarking flow and is useful beyond this specific solver case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100