Sorting results is an ergonomic problem

Open
#167 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
javascript, node.js
Domain
tooling

Research direction

Start by locating the text reporter and the sorting logic that produces the shown benchmark output. Compare the actual and preferred examples; done means results preserve their scheduled order in both the summary and detailed output.

Written by the indexing model from the issue text.

Description

enhancement

The manner in which I utilize bench-node particularly amplifies this effect,

actual

Summary (vs. baseline):
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest   (baseline)
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current  (1.04x faster)
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk    (1.04x faster)

registry ⇒ metrics() 6 x 2 ⇒ latest           x 2,530 ops/sec (41 runs sampled) min..max=(204.50us...429.13us)
registry ⇒ metrics() 6 x 2 ⇒ trunk            x 2,630 ops/sec (41 runs sampled) min..max=(199.25us...400.58us)
registry ⇒ metrics() 6 x 2 ⇒ current          x 2,598 ops/sec (41 runs sampled) min..max=(207.04us...428.31us)

These tests were scheduled in [latest, trunk, current] order. As you can see, one of the sets of output doesn't preserve this order, making reading comprehension a challenge, particularly during a CI run where scrolling is happening. This behavior is built into the text reporter, which sorts the results by runtime. From the git history I suspect intentionally, but I believe this is an ergonomics flaw in the code.

When you're running one test this isn't much of a problem. When you're running (k x n) tests and comparing each k set of results to each other instead of the other more than the other n results (a little of both happens of course), then it becomes very difficult to scan. Misreading leads to taking the wrong corrective actions, or no action at all. And in a CI context that is problematic.

Preferred:

Summary (vs. baseline):
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest   (baseline)
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk    (1.04x faster)
  registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current  (1.04x faster)

The simple examples don't bear this out because the number of results in any single one is 2-3. But when you're looking at 180 batches of 3-5 tests each, then operator error becomes a substantial problem, and any features of the code that hamper reading comprehension begin to look more like bugs instead.

Dominant language
JavaScript
Stars
193
Forks
19
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from RafaelGSS/bench-node

All issues in RafaelGSS/bench-node

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.