JuliaCI / JuliaCI/BenchmarkTools.jl

Repeated `@btime` with different setup behaves bad

Open
#403 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

function sumN(N)
    s = zero(N)
    for i in 1:N
        s += i
    end
    s
end

julia> @btime sumN(N) setup=(N=1_000_000)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_000e0)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_001e0)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(N) setup=(N=1_000_001)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(M) setup=(M=1_000_001)
  1.993 ns (0 allocations: 0 bytes)
500001500001

The Float64 trial should run slowly, and return a Float64, whereas the Int trial should run in a couple of ns and return an Int. Is there some caching of the setup going on, which uses == instead of ===?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the repeated @btime commands from the issue, including the Int and Float64 setup values and the M variable case. Trace how @btime handles repeated setup expressions and compare values using == versus ===. Done means distinct setup types are benchmarked correctly and the reported result preserves the expected numeric type.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.