JuliaCI / JuliaCI/BenchmarkTools.jl
Restructuring a benchmark group breaks loading tuning parameters and judging
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 684
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
I use PkgBenchmark.jl to run my benchmarks in CI. I recently decided to decompose one of my benchmarks into a few benchmarks. So, I originally had an entry in a BenchmarkGroup that was a single benchmark, but the new version instead has a nested BenchmarkGroup for that entry. This caused MethodErrors to be thrown in the BenchmarkTools.jl code, both when loading the tuning parameters and when judging.
ERROR: MethodError: no method matching loadparams!(::BenchmarkTools.Benchmark, ::BenchmarkGroup, ::Symbol, ::Symbol)
Closest candidates are:
loadparams!(::BenchmarkTools.Benchmark, !Matched::BenchmarkTools.Parameters, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/execution.jl:25
loadparams!(!Matched::BenchmarkGroup, ::BenchmarkGroup, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:172
loadparams!(!Matched::BenchmarkTools.Parameters, !Matched::BenchmarkTools.Parameters, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/parameters.jl:96
Stacktrace:
[1] loadparams!(::BenchmarkGroup, ::BenchmarkGroup, ::Symbol, ::Vararg{Symbol}) (repeats 3 times)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:174
[2] __runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:279
[3] (::PkgBenchmark.var"#30#31"{String, String, String, Bool, @NamedTuple{verbose::Bool}})()
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:264
[4] with_logstate(f::Function, logstate::Any)
@ Base.CoreLogging ./logging.jl:515
[5] with_logger
@ ./logging.jl:627 [inlined]
[6] _runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool}, logger_factory_path::Tuple{String, String, Symbol})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:263
[7] top-level scope
@ none:2
ERROR: LoadError: MethodError: no method matching judge(::BenchmarkTools.BenchmarkGroup, ::BenchmarkTools.TrialEstimate)
Closest candidates are:
judge(::BenchmarkTools.BenchmarkGroup, !Matched::BenchmarkTools.BenchmarkGroup...; kwargs...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
judge(!Matched::BenchmarkTools.TrialEstimate, ::BenchmarkTools.TrialEstimate; kwargs...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/trials.jl:231
Stacktrace:
[1] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{Any})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[2] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[3] mapvals
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[4] #judge#24
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[5] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
[6] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[7] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[8] mapvals
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[9] #judge#24
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[10] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
[11] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[12] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[13] mapvals
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[14] #judge#24
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[15] judge
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128 [inlined]
[16] judge(target::BenchmarkResults, baseline::BenchmarkResults, f::typeof(minimum); judgekwargs::Dict{Any, Any})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:50
[17] judge
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49 [inlined]
[18] judge(target::BenchmarkResults, baseline::BenchmarkResults)
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49
[19] top-level scope
@ /nobackup2/gitlab_runner/zDyhf_w4/0/Genesis/Genesis.jl/benchmark/run_ci.jl:24
Could the logic be made more robust to handle a restructuring of benchmarks, similar to what happens if a benchmark is added or removed?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/groups.jl and execution.jl, following the loadparams! and judge methods named in the stack traces. Reproduce a BenchmarkGroup entry changing from a Benchmark to a nested BenchmarkGroup, then trace both tuning-parameter loading and comparison. Done means both operations handle that restructuring without MethodErrors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100