JuliaCI / JuliaCI/BenchmarkTools.jl

log(frequency) in histograms confusing

Open
#237 16 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

While I was benchmarking two very similar functions I noticed the histograms looked very different, which surprised me. This was because the second run switched to a log display of the histogram.
image
(I overwrote the function so they share the same name)

Having the option of a log histogram means I need to check what histogram I'm getting every time. I personally have also trouble reading a log histogram without thinking through what that actually means for my numbers. If one time dominates the histogram I would prefer to see it that way -- I assume the histogram is supposed to be a summary and a single dominant bar would be a good summary in that case. Therefore I would suggest always showing the histogram against the frequency, maybe with an option to display the log histogram.

If the behaviour should stay the same I would like to see an option to pick the histogram type so I can at least compare histograms of measurements easily.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the histogram display used for benchmark results and compare its current frequency and log behavior with the examples in the issue. Decide whether the default should remain unchanged or support an explicit frequency/log choice, then verify that similar benchmark runs can be compared consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.