bencheeorg / bencheeorg/benchee
Odd performance characteristics of using inputs vs. not using inputs
- Dominant language
- Elixir
- Stars
- 1.5k
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
Reworking the README I came upon an interesting performance difference. So let's do our standard example first:
```elixir
list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end
Benchee.run(
%{
"flat_map" => fn -> Enum.flat_map(list, map_fun) end,
"map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten() end
},
# inputs: %{"lol" => Enum.to_list(1..10_000) },
formatters: [{Benchee.Formatters.Console, extended_statistics: true}]
)
```
results in:
```
(...)
Name ips average deviation median 99th %
flat_map 2.37 K 421.36 μs ±14.80% 406.32 μs 725.85 μs
map.flatten 1.23 K 812.20 μs ±18.43% 770.32 μs 1244.07 μs
Comparison:
flat_map 2.37 K
map.flatten 1.23 K - 1.93x slower +390.84 μs
Extended statistics:
Name minimum maximum sample size mode
flat_map 345.36 μs 1199.50 μs 11.84 K 406.20 μs
map.flatten 520.54 μs 1552.89 μs 6.14 K 765.82 μs, 765.75 μs
```
Now let's switch it up slightly and hand the list in through `inputs`:
```elixir
# list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end
Benchee.run(
%{
"flat_map" => fn list -> Enum.flat_map(list, map_fun) end,
"map.flatten" => fn list -> list |> Enum.map(map_fun) |> List.flatten() end
},
inputs: %{"lol" => Enum.to_list(1..10_000) },
formatters: [{Benchee.Formatters.Console, extended_statistics: true}]
)
```
results in:
```
Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Number of Available Cores: 8
Available memory: 15.61 GB
Elixir 1.8.1
Erlang 21.2.7
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: lol
Estimated total run time: 14 s
Benchmarking flat_map with input lol...
Benchmarking map.flatten with input lol...
##### With input lol #####
Name ips average deviation median 99th %
flat_map 2.37 K 421.27 μs ±9.58% 410.08 μs 663.05 μs
map.flatten 1.63 K 612.90 μs ±12.73% 626.79 μs 900.98 μs
Comparison:
flat_map 2.37 K
map.flatten 1.63 K - 1.45x slower +191.63 μs
Extended statistics:
Name minimum maximum sample size mode
flat_map 353.47 μs 1068.95 μs 11.84 K 409.76 μs
map.flatten 519.13 μs 1235.24 μs 8.13 K 519.95 μs
```
All of a sudden **average and median** for `map.flatten` are a lot better (~200μs better). The performance of `flat_map` stays about the same :woman_shrugging:
I tried this multiple times. It's no coincedence, the question is _why_ - if anything I'd have thought that the version without `inputs` would be faster not the other way around.
Trying to summon @michalmuskala who maybe has some compiler input :man_shrugging:
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the README benchmark examples and the Benchee.run handling of inputs, comparing the two reported runs and their output. Reproduce the difference using the shown Elixir, Erlang, and hardware details, then trace why passing inputs changes map.flatten performance. Done means the cause is explained and any required behavior or documentation change is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100