JuliaCI / JuliaCI/BenchmarkTools.jl

Default parameter option `evals_set` is not documented

Open
#374 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

Hi there,

maybe I miss something completely, but I do not understand the meaning of samples and evals for the benchmarks.

If I run this code in REPL

using BenchmarkTools
BenchmarkTools.DEFAULT_PARAMETERS.samples = 1
BenchmarkTools.DEFAULT_PARAMETERS.evals = 1

foo_calls = 0

function foo()
  global foo_calls += 1
end

b = @benchmark foo()

@show foo_calls

b

it should, as I understood the documentation, evaluate foo once. However, it produces

foo_calls = 501494
BenchmarkTools.Trial: 1 sample with 993 evaluations.
 Single result which took 37.263 ns (0.00% GC) to evaluate,
 with a memory estimate of 16 bytes, over 1 allocations.

Why does it evaluate foo 993 times while it was called in reality 501494 times?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with BenchmarkTools.DEFAULT_PARAMETERS and the @benchmark entry point shown in the reproduction. Read the existing benchmark documentation around samples and evals, then document evals_set and clarify why the displayed evaluation count differs from foo_calls. Done means the parameter behavior and the observed output are explained in the documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.