janestreet / janestreet/core_bench

First benchmark is slower than the rest

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
forwarded-to-js-devs
Dominant language
OCaml
Stars
57
Forks
15
PR merge metrics
No merged PRs in 30d

Description

I've noticed that when running benchmarks, the first one that is specified is slower than the rest, even when the same function with the same inputs is benchmarked repeatedly.

Here is a simple example:

```ocaml
open! Core

let rec fib n = if n < 2 then n else fib (n - 1) + fib (n - 2)

let () =
let bench name f = Core_bench.Bench.Test.create ~name f in
Command_unix.run
(Core_bench.Bench.make_command
[
bench "fib 1" (fun () -> fib 10);
bench "fib 2" (fun () -> fib 10);
bench "fib 3" (fun () -> fib 10);
])
```

And the result of running `time dune exec ./bench/a.exe -- -ascii` with a couple different values for fib to see if the behavior changes when shorter or longer running functions are being benched.

```
Name Time/Run Percentage
------- ---------- ------------
fib 1 30.12ns 100.00%
fib 2 18.49ns 61.39%
fib 3 18.41ns 61.11%

Name Time/Run Percentage
------- ---------- ------------
fib 1 317.15ns 100.00%
fib 2 216.46ns 68.25%
fib 3 215.50ns 67.95%

Name Time/Run Percentage
------- ---------- ------------
fib 1 35.66us 100.00%
fib 2 26.90us 75.44%
fib 3 26.97us 75.63%

Name Time/Run Percentage
------- ---------- ------------
fib 1 4.42ms 100.00%
fib 2 3.31ms 74.88%
fib 3 3.32ms 75.00%
```

Is there something wrong with the way I'm setting up the benchmarks that could be causing this?

Contributor guide

Open the contributing guide

Research direction

Reproduce the example using Core_bench.Bench.Test.create, Core_bench.Bench.make_command, and the dune exec ./bench/a.exe -- -ascii command. Compare repeated identical benchmarks in different positions and inspect the benchmark command entry point to determine whether ordering affects measurements. Done means explaining the cause and correcting the behavior or documenting the expected setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.