fastruby / fastruby/fast-ruby

Some benchmarks are measured with wrong approach

Abierto
#96 17 comentarios 9 reacciones 0 asignados Ver en GitHub
General
Lenguaje dominante
Ruby
Estrellas
5.7k
Forks
371
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

**tl;dr** When we measure small things we have to remove all overheads which have impact. Wrapping with a method and block call is quite big overhead for most of benchmarks in this repository.

Suppose you want compare performance of "2 + 3" vs "2 \* 3" calls. If you do it with a approach used in this repository you will get this results:

``` ruby
require "benchmark/ips"

def slow
2 * 2
end

def fast
2 + 2
end

Benchmark.ips do |x|
x.report("2 * 2") { slow }
x.report("2 + 2") { fast }
x.compare!
end
```

(simplified output)

```
Comparison:
2 + 2: 8304760.0 i/s
2 * 2: 7535516.6 i/s - 1.10x slower
```

But there is one problem. Calling a method + surrounding block (`{ slow }`) has bigger overhead than calling `Fixnum#+` or `Fixnum#*` itself. Thea easiest way to observe it is to repeat benchmarked operations in one call. Like this:

``` ruby
require "benchmark/ips"

def slow
2*2; 2*2; 2*2; 2*2; 2*2; 2*2; 2*2; 2*2; 2*2; 2*2;
end

def fast
2+2; 2+2; 2+2; 2+2; 2+2; 2+2; 2+2; 2+2; 2+2; 2+2;
end

Benchmark.ips do |x|
x.report("2 * 2") { slow }
x.report("2 + 2") { fast }
x.compare!
end
```

```
Comparison:
2 + 2: 4680545.3 i/s
2 * 2: 3468681.3 i/s - 1.35x slower
```

See how results changed? Writing our benchmarks in this way would be quite problematic. Fortunately benchmark-ips gem has answer for that. `Benchmark::IPS::Job#report` method allows to pass string which will be compiled before benchmark is run. Passing right string allows to measure it properly:

``` ruby
require "benchmark/ips"

Benchmark.ips do |x|
x.report("2 * 2", "2 * 2;" * 1_000)
x.report("2 + 2", "2 + 2;" * 1_000)
x.compare!
end
```

```
Comparison:
2 + 2: 91567.5 i/s
2 * 2: 57994.4 i/s - 1.58x slower
```

This is greatly explained here: [https://docs.omniref.com/ruby/2.2.1/symbols/Benchmark/bm#annotation=4095926&line=182](https://docs.omniref.com/ruby/2.2.1/symbols/Benchmark/bm#annotation=4095926&line=182)

How does this affect fast-ruby benchmarks? All benchmarks that call small things are flawed. One of them is `Array#length vs Array#size vs Array#count` benchmark. Here is the [original code](https://github.com/JuanitoFatas/fast-ruby/blob/ca29eedcdc55c7fa6af00e70fbb58cdd6f6c7bea/code/array/length-vs-size-vs-count.rb) and result obtained on my computer:

``` ruby
require 'benchmark/ips'

ARRAY = [*1..100]

Benchmark.ips do |x|
x.report("Array#length") { ARRAY.length }
x.report("Array#size") { ARRAY.size }
x.report("Array#count") { ARRAY.count }
x.compare!
end
```

```
Comparison:
Array#size: 8679483.2 i/s
Array#length: 8664450.7 i/s - 1.00x slower
Array#count: 7237299.5 i/s - 1.20x slower
```

The same benchmark measure with described approach gives different numbers:

``` ruby
require 'benchmark/ips'

ARRAY = [*1..100]

Benchmark.ips do |x|
x.report("Array#length", "ARRAY.length;" * 1_000)
x.report("Array#size", "ARRAY.size;" * 1_000)
x.report("Array#count", "ARRAY.count;" * 1_000)
x.compare!
end
```

```
Comparison:
Array#size: 113902.4 i/s
Array#length: 113655.9 i/s - 1.00x slower
Array#count: 28753.4 i/s - 3.96x slower
```

**Difference: 1.20x slower vs 3.96x slower.**

My guess is that it affects most of benchmarks.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.