ruby / ruby/benchmark

Make benchmarks nicer to read by default, via the commandline.

Open
#23 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
213
Forks
25
Avg merge
4d 4h
Merged PRs (30d)
2

Description

The ruby benchmark ecosystem is quite nice; it reports things, aka what is faster
aka more efficient.

Take the following old code I have:

# Interpolation is slowest, interestingly enough.
require 'benchmark'

N_TIMES = 5_000_000

a = 'foo'.freeze
b = 'bar'.freeze

Benchmark.bmbm(30) { |entry|
  entry.report('(1) interpol: #{a}#{b}  ') { N_TIMES.times { "#{a}#{b}" } }
  entry.report('(2) plus:     a+b       ') { N_TIMES.times { a + b      } }
  entry.report('(3) append:   a.dup << b') { N_TIMES.times { a.dup << b } }
}

Results are:

Rehearsal ------------------------------------------------------------------
(1) interpol: #{a}#{b}           0.534171   0.005792   0.539963 (  0.539974)
(2) plus:     a+b                0.419223   0.000000   0.419223 (  0.419227)
(3) append:   a.dup << b         0.413437   0.003269   0.416706 (  0.416715)
--------------------------------------------------------- total: 1.375892sec

                                     user     system      total        real
(1) interpol: #{a}#{b}           0.544134   0.006647   0.550781 (  0.550799)
(2) plus:     a+b                0.417118   0.000000   0.417118 (  0.417120)
(3) append:   a.dup << b         0.421809   0.013338   0.435147 (  0.435174)

Now from these results, it seems interpolation is slower than the other two
methods. I take it the most important is the very right value in () parens.

But I am not sure if I interprete this correctly.

Would it be possible for ruby to add a "verbose" output result, aka a
human-readable summary? With that I mean something like "did you
mean" gem, but adjusted to benchmark interpretation.

Something like:

"The above results indicate that interpolation (1) is fastest, whereas the
other two methods are -15% (2) and -14% (3) slower."

The above sentence should just indicate what I am trying to convey.

I'd like for benchmark to become more useful to normal people without
a PhD. Just like the did-you-mean gem, which is super-simple and
suggests what could be wrong. Sometimes this is not quite correct,
but most of the time did-you-mean is super-correct and easy to
understand. Benchmark evaluation are not so easy to understand
and I'd like a dedicated "report" subsystem there, that is human-readable.
Aka the output should be simpler, and using that subsystem in the
benchmarks module should ALSO be super-simple, even the default;
or, if not the default, then a simple option, a simple API, to call it,
such as:

Benchmark.bmbm(30, :be_verbose) { |entry|

Or API-wise:

Benchmark.verbose_bmbm(30) { |entry|
Benchmark.bmbm_verbose(30) { |entry|

Or some other, somewhat similar name.

If added then this should also be documented, at the least with one
usage example, so people can quickly start incorporating this into
their own benchmarks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing Benchmark.bmbm API and its current report output, then review the benchmark documentation. Define how a command-line option or API should produce a human-readable comparison, and document it with a usage example. Done means the new reporting path is usable without manually interpreting the timing columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.