bencheeorg / bencheeorg/benchee

Ability to set a "reference" job

Open
#179 2 comments 3 reactions 0 assignees View on GitHub
help wanted
Dominant language
Elixir
Stars
1.5k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Right now when benchmarks are compared, they are always compared to the fastest run. I think it would be useful in some situations to set a "reference" run and compare all the other runs to that.

One example would be the benchmark suite of `jason`, where I'd like to see all the other encoders reported relative to Jason itself, even though `jiffy` is the fastest most of the time. Right now, I have to do the additional computations manually, to get the desired numbers, that I can use to annoy people convincing them how fast Jason is 😉.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Trace the benchmark comparison path that selects the fastest run, then inspect how runs and encoders are represented; done means a chosen reference run can be used for all comparison results, including the Jason benchmark example.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.