JuliaCI / JuliaCI/BenchmarkTools.jl
The metrics for the mean and standard deviation of (percentage-wise) GC are not consistent with each other
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 684
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
I was trying to understand better how each number in the printed information of a BenchmarkTools.Trial is derived. I noticed an inconsistency between the mean and the standard deviation (SD) computed for the ratioed GC time. Consider the following case:
julia> using BenchmarkTools
julia> using Statistics
julia> function foo(a::Int)
v = rand(3,3)
(v.^a) |> sum
end
foo (generic function with 1 method)
julia> b = @benchmark foo(2)
BenchmarkTools.Trial: 10000 samples with 978 evaluations per sample.
Range (min … max): 69.632 ns … 1.398 μs ┊ GC (min … max): 0.00% … 87.32%
Time (median): 71.268 ns ┊ GC (median): 0.00%
Time (mean ± σ): 78.708 ns ± 44.711 ns ┊ GC (mean ± σ): 6.78% ± 10.74%
█▄ ▁
███▆▅▅▅▄▄▅▄▆▇▅▄▅▇▆▃▄▃▁▁▄▁▁▁▁▃▁▁▁▁▆▆▄▆▆▃▄▁▄▄▁▁▄▃▁▁▁▁▁▁▁▄▆▇██ █
69.6 ns Histogram: log(frequency) by time 315 ns <
Memory estimate: 288 bytes, allocs estimate: 4.
The SD is computed w.r.t. the ratios of GC time over the total time for each sample:
julia> isapprox(0.1074, Statistics.std(b.gctimes ./ b.times), atol=1e-4)
true
which can also be verified through the source code.
However, for the mean, instead of the Mean of the Ratios (MoR), the printed information showed the Ratio of the Means (RoM):
julia> isapprox(0.0678, Statistics.mean(b.gctimes ./ b.times), atol=1e-4)
false
julia> isapprox(0.0678, Statistics.mean(b.gctimes) / Statistics.mean(b.times), atol=1e-4)
true
The corresponding source code verification can be found here.
Both MoR and RoM have statistical significance. However, if we want to print the mean and SD for the GC time together, like BenchmarkTools.jl's current printing format, I think they should be consistently set w.r.t. the same random variable: the ratioed GC time (GC time / total time per sample).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading src/trials.jl around the linked lines 409-411 and reproduce the reported Julia example with BenchmarkTools and Statistics. Compare the mean and standard-deviation calculations for GC time, then make the printed metrics consistent for the selected ratio and verify the example's output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100