DynamoRIO / DynamoRIO/drmemory
Slow execution of the Cicro benchmarks
- Dominant language
- C
- Stars
- 2.7k
- Forks
- 290
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I tried measuring Dr. Memory's performance on the [Cicro benchmarks](https://github.com/jku-ssw/cicro-benchmarks). I was surprised that the performance overhead of Dr. Memory was much higher than for Valgrind. For example. Dr. Memory timed out after 2 hours for the `adamdunkels-ubasic` benchmark, while Memcheck executed it in about 2 minutes.
ubasic Output for Dr. Memory:
```
[2019-04-17 02:21:33,395] [ INFO ] benchmark: "adamdunkels-ubasic:drmemory-O3" (bench.py:380)
[2019-04-17 04:30:26,312] [ ERROR ] Something went wrong while running the benchmark: "adamdunkels-ubasic:drmemory-O3" (bench.py:434)
Traceback (most recent call last):
File "/benchmarks/tools/bench.py", line 384, in execute_single_runtime
stdout, stderr, exit_code = exec_func(harness_path, self.testdir, exec_args, **exec_kwargs)
File "", line 112, in drmemory_executor
File "", line 95, in execute_binary_analysis_tool
File "/usr/lib/python3.6/subprocess.py", line 843, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.6/subprocess.py", line 1515, in _communicate
self._check_timeout(endtime, orig_timeout)
File "/usr/lib/python3.6/subprocess.py", line 871, in _check_timeout
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['/DrMemory-Linux-2.1.0-1/bin64/drmemory', '-exit_code_if_errors', '1', '--', '/benchmarks/benchmarks/adamdunkels-ubasic_test', '--output=json']' timed out after 7200 seconds
```
ubasic Output for Valgrind:
```
[2019-04-17 15:35:49,631] [ INFO ] benchmark: "adamdunkels-ubasic:valgrind-O3" (bench.py:380)
[2019-04-17 15:37:03,685] [ WARNING ] benchmark harness had some output on stderr:
==4594== Memcheck, a memory error detector
==4594== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==4594== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==4594== Command: /benchmarks/benchmarks/adamdunkels-ubasic_test --output=json
==4594==
==4594==
==4594== HEAP SUMMARY:
==4594== in use at exit: 168 bytes in 7 blocks
==4594== total heap usage: 71,403 allocs, 71,396 frees, 1,719,007 bytes allocated
==4594==
==4594== LEAK SUMMARY:
==4594== definitely lost: 0 bytes in 0 blocks
==4594== indirectly lost: 0 bytes in 0 blocks
==4594== possibly lost: 0 bytes in 0 blocks
==4594== still reachable: 168 bytes in 7 blocks
==4594== suppressed: 0 bytes in 0 blocks
==4594== Rerun with --leak-check=full to see details of leaked memory
==4594==
==4594== For counts of detected and suppressed errors, rerun with: -v
==4594== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
(bench.py:392)
```
The issue can be reproduced with the following commands:
```
git clone https://github.com/jku-ssw/cicro-benchmarks/
cd cicro-benchmarks/docker && make build_BENCHMARK_BASE build_BENCHMARK_HARNESS build_BENCHMARK_DRMEMORY && cd ..
cd paper && make data/drmemory.json
```
To run with Memcheck, you can use `make data/memcheck.json`. Note that both Dr. Memory and Memcheck are executed in a Docker container. The `Dockerfile` used to build Dr. Memory is located [here](https://github.com/jku-ssw/cicro-benchmarks/blob/5839268d1f880ed72954d3a2712315c6b5a4a7ec/docker/BENCHMARK_DRMEMORY/Dockerfile). Dr. Memory is invoked [here](https://github.com/jku-ssw/cicro-benchmarks/blob/5839268d1f880ed72954d3a2712315c6b5a4a7ec/configs/valgrind.py#L112).
Do you have any ideas on what the issue could be or how to isolate the problem?
Contributor guide
Assessment
This issue has not been assessed yet.