DynamoRIO / DynamoRIO/drmemory

Slow execution of the Cicro benchmarks

Open
#2,191 3 comments 0 reactions 0 assignees View on GitHub
OpSys-x64 Performance
Dominant language
C
Stars
2.7k
Forks
290
PR merge metrics
No merged PRs in 30d

Description

Hi, I tried measuring Dr. Memory's performance on the [Cicro benchmarks](https://github.com/jku-ssw/cicro-benchmarks). I was surprised that the performance overhead of Dr. Memory was much higher than for Valgrind. For example. Dr. Memory timed out after 2 hours for the `adamdunkels-ubasic` benchmark, while Memcheck executed it in about 2 minutes.

ubasic Output for Dr. Memory:
```
[2019-04-17 02:21:33,395] [ INFO ] benchmark: "adamdunkels-ubasic:drmemory-O3" (bench.py:380)
[2019-04-17 04:30:26,312] [ ERROR ] Something went wrong while running the benchmark: "adamdunkels-ubasic:drmemory-O3" (bench.py:434)
Traceback (most recent call last):
File "/benchmarks/tools/bench.py", line 384, in execute_single_runtime
stdout, stderr, exit_code = exec_func(harness_path, self.testdir, exec_args, **exec_kwargs)
File "", line 112, in drmemory_executor
File "", line 95, in execute_binary_analysis_tool
File "/usr/lib/python3.6/subprocess.py", line 843, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.6/subprocess.py", line 1515, in _communicate
self._check_timeout(endtime, orig_timeout)
File "/usr/lib/python3.6/subprocess.py", line 871, in _check_timeout
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['/DrMemory-Linux-2.1.0-1/bin64/drmemory', '-exit_code_if_errors', '1', '--', '/benchmarks/benchmarks/adamdunkels-ubasic_test', '--output=json']' timed out after 7200 seconds
```

ubasic Output for Valgrind:
```
[2019-04-17 15:35:49,631] [ INFO ] benchmark: "adamdunkels-ubasic:valgrind-O3" (bench.py:380)
[2019-04-17 15:37:03,685] [ WARNING ] benchmark harness had some output on stderr:
==4594== Memcheck, a memory error detector
==4594== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==4594== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==4594== Command: /benchmarks/benchmarks/adamdunkels-ubasic_test --output=json
==4594==
==4594==
==4594== HEAP SUMMARY:
==4594== in use at exit: 168 bytes in 7 blocks
==4594== total heap usage: 71,403 allocs, 71,396 frees, 1,719,007 bytes allocated
==4594==
==4594== LEAK SUMMARY:
==4594== definitely lost: 0 bytes in 0 blocks
==4594== indirectly lost: 0 bytes in 0 blocks
==4594== possibly lost: 0 bytes in 0 blocks
==4594== still reachable: 168 bytes in 7 blocks
==4594== suppressed: 0 bytes in 0 blocks
==4594== Rerun with --leak-check=full to see details of leaked memory
==4594==
==4594== For counts of detected and suppressed errors, rerun with: -v
==4594== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
(bench.py:392)
```

The issue can be reproduced with the following commands:
```
git clone https://github.com/jku-ssw/cicro-benchmarks/
cd cicro-benchmarks/docker && make build_BENCHMARK_BASE build_BENCHMARK_HARNESS build_BENCHMARK_DRMEMORY && cd ..
cd paper && make data/drmemory.json
```

To run with Memcheck, you can use `make data/memcheck.json`. Note that both Dr. Memory and Memcheck are executed in a Docker container. The `Dockerfile` used to build Dr. Memory is located [here](https://github.com/jku-ssw/cicro-benchmarks/blob/5839268d1f880ed72954d3a2712315c6b5a4a7ec/docker/BENCHMARK_DRMEMORY/Dockerfile). Dr. Memory is invoked [here](https://github.com/jku-ssw/cicro-benchmarks/blob/5839268d1f880ed72954d3a2712315c6b5a4a7ec/configs/valgrind.py#L112).

Do you have any ideas on what the issue could be or how to isolate the problem?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.