Sorting results is an ergonomic problem
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- javascript, node.js
- Domain
- tooling
Research direction
Start by locating the text reporter and the sorting logic that produces the shown benchmark output. Compare the actual and preferred examples; done means results preserve their scheduled order in both the summary and detailed output.
Written by the indexing model from the issue text.
Description
The manner in which I utilize bench-node particularly amplifies this effect,
actual
Summary (vs. baseline):
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest (baseline)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current (1.04x faster)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk (1.04x faster)
registry ⇒ metrics() 6 x 2 ⇒ latest x 2,530 ops/sec (41 runs sampled) min..max=(204.50us...429.13us)
registry ⇒ metrics() 6 x 2 ⇒ trunk x 2,630 ops/sec (41 runs sampled) min..max=(199.25us...400.58us)
registry ⇒ metrics() 6 x 2 ⇒ current x 2,598 ops/sec (41 runs sampled) min..max=(207.04us...428.31us)
These tests were scheduled in [latest, trunk, current] order. As you can see, one of the sets of output doesn't preserve this order, making reading comprehension a challenge, particularly during a CI run where scrolling is happening. This behavior is built into the text reporter, which sorts the results by runtime. From the git history I suspect intentionally, but I believe this is an ergonomics flaw in the code.
When you're running one test this isn't much of a problem. When you're running (k x n) tests and comparing each k set of results to each other instead of the other more than the other n results (a little of both happens of course), then it becomes very difficult to scan. Misreading leads to taking the wrong corrective actions, or no action at all. And in a CI context that is problematic.
Preferred:
Summary (vs. baseline):
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest (baseline)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk (1.04x faster)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current (1.04x faster)
The simple examples don't bear this out because the number of results in any single one is 2-3. But when you're looking at 180 batches of 3-5 tests each, then operator error becomes a substantial problem, and any features of the code that hamper reading comprehension begin to look more like bugs instead.
- Dominant language
- JavaScript
- Stars
- 193
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from RafaelGSS/bench-node
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
RafaelGSS/bench-node#168 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
RafaelGSS/bench-node#150 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
RafaelGSS/bench-node#138 · 4 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
RafaelGSS/bench-node#130 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
RafaelGSS/bench-node#121 · 3 comments ·
All issues in RafaelGSS/bench-node
Similar issues
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
github/gh-aw-firewall#8816 ·
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100