oxc-project / oxc-project/backlog

Add wallclock benchmarks

Open
#5 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
7
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Problem

CodSpeed is good, but has some anomalies.

In particular:

  1. All system calls (e.g. system allocator) are "free" on Codspeed's measures.
  2. They say it doesn't take into account the branch predictor.

Mispredicted branches can be a significant perf hit, which we're failing to measure. In particular, I have some ideas to replace branching in lexer with straight-line code (https://github.com/oxc-project/backlog/issues/108). I suspect this could be a significant gain, but it won't register on current CodSpeed benchmarks - so we can't evaluate this at present.

Possible solution

Introduce wallclock benchmarks (not run with Valgrind) in addition to the existing benchmarks.

How?

Can use the same hack I wrote to run NAPI benchmarks as normal wallclock benchmarks and get the results into CodSpeed.

An improvement would be if it's possible to synthesize fake .out files to send to CodSpeed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing hack used to run NAPI benchmarks as normal wallclock benchmarks and how current CodSpeed benchmarks produce results. Determine whether wallclock runs can be integrated directly or require synthetic .out files. Done means wallclock benchmarks run outside Valgrind and their results reach CodSpeed alongside the existing measurements.

Written by the indexing model from the issue text.

Assessment

Domain
performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.