bencheeorg / bencheeorg/benchee

Odd performance characteristics of using inputs vs. not using inputs

Open
#282 6 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Elixir
Stars
1.5k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

Reworking the README I came upon an interesting performance difference. So let's do our standard example first:

```elixir
list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end

Benchee.run(
%{
"flat_map" => fn -> Enum.flat_map(list, map_fun) end,
"map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten() end
},
# inputs: %{"lol" => Enum.to_list(1..10_000) },
formatters: [{Benchee.Formatters.Console, extended_statistics: true}]
)
```

results in:

```
(...)

Name ips average deviation median 99th %
flat_map 2.37 K 421.36 μs ±14.80% 406.32 μs 725.85 μs
map.flatten 1.23 K 812.20 μs ±18.43% 770.32 μs 1244.07 μs

Comparison:
flat_map 2.37 K
map.flatten 1.23 K - 1.93x slower +390.84 μs

Extended statistics:

Name minimum maximum sample size mode
flat_map 345.36 μs 1199.50 μs 11.84 K 406.20 μs
map.flatten 520.54 μs 1552.89 μs 6.14 K 765.82 μs, 765.75 μs
```

Now let's switch it up slightly and hand the list in through `inputs`:

```elixir
# list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end

Benchee.run(
%{
"flat_map" => fn list -> Enum.flat_map(list, map_fun) end,
"map.flatten" => fn list -> list |> Enum.map(map_fun) |> List.flatten() end
},
inputs: %{"lol" => Enum.to_list(1..10_000) },
formatters: [{Benchee.Formatters.Console, extended_statistics: true}]
)
```

results in:

```
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Number of Available Cores: 8
Available memory: 15.61 GB
Elixir 1.8.1
Erlang 21.2.7

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: lol
Estimated total run time: 14 s

Benchmarking flat_map with input lol...
Benchmarking map.flatten with input lol...

##### With input lol #####
Name ips average deviation median 99th %
flat_map 2.37 K 421.27 μs ±9.58% 410.08 μs 663.05 μs
map.flatten 1.63 K 612.90 μs ±12.73% 626.79 μs 900.98 μs

Comparison:
flat_map 2.37 K
map.flatten 1.63 K - 1.45x slower +191.63 μs

Extended statistics:

Name minimum maximum sample size mode
flat_map 353.47 μs 1068.95 μs 11.84 K 409.76 μs
map.flatten 519.13 μs 1235.24 μs 8.13 K 519.95 μs
```

All of a sudden **average and median** for `map.flatten` are a lot better (~200μs better). The performance of `flat_map` stays about the same :woman_shrugging:

I tried this multiple times. It's no coincedence, the question is _why_ - if anything I'd have thought that the version without `inputs` would be faster not the other way around.

Trying to summon @michalmuskala who maybe has some compiler input :man_shrugging:

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README benchmark examples and the Benchee.run handling of inputs, comparing the two reported runs and their output. Reproduce the difference using the shown Elixir, Erlang, and hardware details, then trace why passing inputs changes map.flatten performance. Done means the cause is explained and any required behavior or documentation change is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.