oxc-project / oxc-project/backlog
Add wallclock benchmarks
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 7
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Problem
CodSpeed is good, but has some anomalies.
In particular:
- All system calls (e.g. system allocator) are "free" on Codspeed's measures.
- They say it doesn't take into account the branch predictor.
Mispredicted branches can be a significant perf hit, which we're failing to measure. In particular, I have some ideas to replace branching in lexer with straight-line code (https://github.com/oxc-project/backlog/issues/108). I suspect this could be a significant gain, but it won't register on current CodSpeed benchmarks - so we can't evaluate this at present.
Possible solution
Introduce wallclock benchmarks (not run with Valgrind) in addition to the existing benchmarks.
How?
Can use the same hack I wrote to run NAPI benchmarks as normal wallclock benchmarks and get the results into CodSpeed.
An improvement would be if it's possible to synthesize fake .out files to send to CodSpeed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the existing hack used to run NAPI benchmarks as normal wallclock benchmarks and how current CodSpeed benchmarks produce results. Determine whether wallclock runs can be integrated directly or require synthetic .out files. Done means wallclock benchmarks run outside Valgrind and their results reach CodSpeed alongside the existing measurements.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100