google / google/caliper

Provide a better solution for non deterministic allocation behavior

Open
#273 0 comments 0 reactions 0 assignees View on GitHub
component: runner P4 type=addition
Dominant language
Java
Stars
818
Forks
112
Avg merge
12m
Merged PRs (30d)
5

Description

```
Currently caliper will aggressively check for non deterministic allocation
behavior and fail the measurement when non determinism is detected.

This can be partially worked around by setting
instrument.allocation.options.trackAllocations=false
which simply reduces the amount of data that caliper tracks and thus limits the
kinds of non determinism that can be detected.

On the one hand a non deterministic benchmark is a bad thing since it makes
your measurements suspect so users should try to eliminate it (i.e. make the
benchmark perform the exact same amount of work on each rep). However, not all
non-determinism (of allocation behavior) can be eliminated. For example:

In multithreaded benchmarks, lock contention will cause allocations and is
inherently non deterministic.
Lazy initialization of data is a common pattern and its possible that not all
such initialization can be triggered in warmup.
and others

Caliper should try to support such use cases.

One proposal would be for the AllocationWorker to be configured to work like
the RuntimeWorker and instead of subtracting a baseline measurement to remove
overheads just run a large number of reps to minimize them. This could be
configurable via something like
instrument.allocation.options.strategy

```

Original issue reported on code.google.com by `lu...@google.com` on 16 Aug 2013 at 4:22

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.