JuliaCI / JuliaCI/BenchmarkTools.jl

Restructuring a benchmark group breaks loading tuning parameters and judging

Open
#396 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

I use PkgBenchmark.jl to run my benchmarks in CI. I recently decided to decompose one of my benchmarks into a few benchmarks. So, I originally had an entry in a BenchmarkGroup that was a single benchmark, but the new version instead has a nested BenchmarkGroup for that entry. This caused MethodErrors to be thrown in the BenchmarkTools.jl code, both when loading the tuning parameters and when judging.

ERROR: MethodError: no method matching loadparams!(::BenchmarkTools.Benchmark, ::BenchmarkGroup, ::Symbol, ::Symbol)
Closest candidates are:
  loadparams!(::BenchmarkTools.Benchmark, !Matched::BenchmarkTools.Parameters, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/execution.jl:25
  loadparams!(!Matched::BenchmarkGroup, ::BenchmarkGroup, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:172
  loadparams!(!Matched::BenchmarkTools.Parameters, !Matched::BenchmarkTools.Parameters, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/parameters.jl:96
Stacktrace:
 [1] loadparams!(::BenchmarkGroup, ::BenchmarkGroup, ::Symbol, ::Vararg{Symbol}) (repeats 3 times)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:174
 [2] __runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool})
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:279
 [3] (::PkgBenchmark.var"#30#31"{String, String, String, Bool, @NamedTuple{verbose::Bool}})()
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:264
 [4] with_logstate(f::Function, logstate::Any)
   @ Base.CoreLogging ./logging.jl:515
 [5] with_logger
   @ ./logging.jl:627 [inlined]
 [6] _runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool}, logger_factory_path::Tuple{String, String, Symbol})
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:263
 [7] top-level scope
   @ none:2
ERROR: LoadError: MethodError: no method matching judge(::BenchmarkTools.BenchmarkGroup, ::BenchmarkTools.TrialEstimate)
Closest candidates are:
  judge(::BenchmarkTools.BenchmarkGroup, !Matched::BenchmarkTools.BenchmarkGroup...; kwargs...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
  judge(!Matched::BenchmarkTools.TrialEstimate, ::BenchmarkTools.TrialEstimate; kwargs...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/trials.jl:231
Stacktrace:
  [1] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{Any})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
  [2] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
  [3] mapvals
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
  [4] #judge#24
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
  [5] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
  [6] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
  [7] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
  [8] mapvals
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
  [9] #judge#24
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
 [10] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
 [11] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
 [12] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
 [13] mapvals
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
 [14] #judge#24
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
 [15] judge
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128 [inlined]
 [16] judge(target::BenchmarkResults, baseline::BenchmarkResults, f::typeof(minimum); judgekwargs::Dict{Any, Any})
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:50
 [17] judge
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49 [inlined]
 [18] judge(target::BenchmarkResults, baseline::BenchmarkResults)
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49
 [19] top-level scope
    @ /nobackup2/gitlab_runner/zDyhf_w4/0/Genesis/Genesis.jl/benchmark/run_ci.jl:24

Could the logic be made more robust to handle a restructuring of benchmarks, similar to what happens if a benchmark is added or removed?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/groups.jl and execution.jl, following the loadparams! and judge methods named in the stack traces. Reproduce a BenchmarkGroup entry changing from a Benchmark to a nested BenchmarkGroup, then trace both tuning-parameter loading and comparison. Done means both operations handle that restructuring without MethodErrors.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.