JuliaCI / JuliaCI/BenchmarkTools.jl

`@btime` reports wrong allocation

Open
#360 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

MWE:

using BenchmarkTools

struct S
    x
end
function (s::S)()
    s.x[1]
end

function f()
    s1 = S(rand(10))
    @time s1()
    @btime $s1()
    println(@allocated s1())
    s2 = S(1)
    @time s2()
    @btime $s2()
    @benchmark $s2()
    println(@allocated s2())
end

f()

The output I see is

  0.000001 seconds
  19.435 ns (1 allocation: 16 bytes)
0
  0.000000 seconds
  18.050 ns (0 allocations: 0 bytes)
0

So when the field is a vector, @btime reports wrong allocation, assuming @time and @allocated are right. This may be related to https://github.com/JuliaLang/julia/issues/53694 but I'm not sure.

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, broadwell)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied MWE with Julia 1.10.2 and BenchmarkTools, comparing @btime and @benchmark results with @time and @allocated for both S cases. Then inspect the implementation of the @btime and @benchmark macros; the work is done when the reported allocation for the vector-field case matches the measured allocation and regression coverage is added.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.