JuliaCI / JuliaCI/BenchmarkTools.jl
Default parameter option `evals_set` is not documented
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 684
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
maybe I miss something completely, but I do not understand the meaning of samples and evals for the benchmarks.
If I run this code in REPL
using BenchmarkTools
BenchmarkTools.DEFAULT_PARAMETERS.samples = 1
BenchmarkTools.DEFAULT_PARAMETERS.evals = 1
foo_calls = 0
function foo()
global foo_calls += 1
end
b = @benchmark foo()
@show foo_calls
b
it should, as I understood the documentation, evaluate foo once. However, it produces
foo_calls = 501494
BenchmarkTools.Trial: 1 sample with 993 evaluations.
Single result which took 37.263 ns (0.00% GC) to evaluate,
with a memory estimate of 16 bytes, over 1 allocations.
Why does it evaluate foo 993 times while it was called in reality 501494 times?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with BenchmarkTools.DEFAULT_PARAMETERS and the @benchmark entry point shown in the reproduction. Read the existing benchmark documentation around samples and evals, then document evals_set and clarify why the displayed evaluation count differs from foo_calls. Done means the parameter behavior and the observed output are explained in the documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100