JuliaCI / JuliaCI/BenchmarkTools.jl
Repeated `@btime` with different setup behaves bad
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 684
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
function sumN(N)
s = zero(N)
for i in 1:N
s += i
end
s
end
julia> @btime sumN(N) setup=(N=1_000_000)
2.424 ns (0 allocations: 0 bytes)
500000500000
julia> @btime sumN(N) setup=(N=1_000_000e0)
2.424 ns (0 allocations: 0 bytes)
500000500000
julia> @btime sumN(N) setup=(N=1_000_001e0)
1.330 ms (0 allocations: 0 bytes)
5.00001500001e11
julia> @btime sumN(N) setup=(N=1_000_001)
1.330 ms (0 allocations: 0 bytes)
5.00001500001e11
julia> @btime sumN(M) setup=(M=1_000_001)
1.993 ns (0 allocations: 0 bytes)
500001500001
The Float64 trial should run slowly, and return a Float64, whereas the Int trial should run in a couple of ns and return an Int. Is there some caching of the setup going on, which uses == instead of ===?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the repeated @btime commands from the issue, including the Int and Float64 setup values and the M variable case. Trace how @btime handles repeated setup expressions and compare values using == versus ===. Done means distinct setup types are benchmarked correctly and the reported result preserves the expected numeric type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100