vitest-dev / vitest-dev/vitest

[5.x] Add `bench.all` to group benchmarks without comparing them

Open
#11,147 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement: pending triage
Dominant language
TypeScript
Stars
17.1k
Forks
2k
Avg merge
1d 22h
Merged PRs (30d)
94

Description

Clear and concise description of the problem

With the recent benchmarking API overhaul, I'd love for there to be a new bench.all API that works similar to bench.compare, but with some notable changes:

  1. The results are sorted by argument order and not by speed
  2. Each run will get the same options, and not do more iterations just because it's faster than others in the group.
Suggested solution

A new bench.all API, similar to bench.compare.

Not sure if tinybench/vitest already does this, but it might be good to have a default of "interleaved" runs, and only do "sequential" benchmarking when configured. This would allow for more real-life use cases when benchmarking I feel.

Alternative

No response

Additional context

Currently, running benchmarks individually will show each benchmark as its own little table, making the results very hard to read/parse.

This issue was opened after a Discord discussion: https://discord.com/channels/917386801235247114/917386801235247117/1545405463972282480

Validations

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing bench.compare API and review the benchmarking behavior discussed for tinybench/vitest. Define how bench.all should preserve argument order, share options and present grouped results, including whether interleaved runs are the default. Done means the new API is covered by tests for ordering, iteration options and readable grouped output.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.