vitest-dev / vitest-dev/vitest
[5.x] Add `bench.all` to group benchmarks without comparing them
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 17.1k
- Forks
- 2k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 94
Description
Clear and concise description of the problem
With the recent benchmarking API overhaul, I'd love for there to be a new bench.all API that works similar to bench.compare, but with some notable changes:
- The results are sorted by argument order and not by speed
- Each run will get the same options, and not do more iterations just because it's faster than others in the group.
Suggested solution
A new bench.all API, similar to bench.compare.
Not sure if tinybench/vitest already does this, but it might be good to have a default of "interleaved" runs, and only do "sequential" benchmarking when configured. This would allow for more real-life use cases when benchmarking I feel.
Alternative
No response
Additional context
Currently, running benchmarks individually will show each benchmark as its own little table, making the results very hard to read/parse.
This issue was opened after a Discord discussion: https://discord.com/channels/917386801235247114/917386801235247117/1545405463972282480
Validations
- Follow our Code of Conduct
- Read the Contributing Guidelines.
- Read the docs.
- Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing bench.compare API and review the benchmarking behavior discussed for tinybench/vitest. Define how bench.all should preserve argument order, share options and present grouped results, including whether interleaved runs are the default. Done means the new API is covered by tests for ordering, iteration options and readable grouped output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100