DynamoRIO / DynamoRIO/dynamorio
Linux: coredump in try_compile might break build
- Dominant language
- C
- Stars
- 3.2k
- Forks
- 629
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 30
Description
While working on the packaging of dynamorio for debian #5153, I stumbled upon the following issue when building dynamorio in a minimal CI system. The reason is a unlucky combination of directory names of dynamorio (top level `core`) and the default coredump file name.
- DynamoRIO contains a directory called `core`
- The default coredump filename is `core`
- During the CMake feature probing (`try_compile`), a coredump happens which creates a file `core`, making it impossible to create the `core` directory later on
Citing `man 5 core`
```
Naming of core dump files
By default, a core dump file is named core, but the
/proc/sys/kernel/core_pattern file (since Linux 2.6 and 2.4.21)
can be set to define a template that is used to name core dump
files. The template can contain % specifiers which are
substituted by the following values when a core file is created:
```
To validate, please check the following files:
```bash
cat /proc/sys/kernel/core_uses_pid
# 0 is problematic
cat /proc/sys/kernel/core_pattern
# core is problematic
```
As a mitigation, enable `core_uses_pid` via `sysctl.d` or simply install `systemd-coredump`.
**/etc/sysctl.d/90-coredump.conf**
```
kernel.core_uses_pid = 1
```
Error pattern:
```
-- Configuring done
CMake Error: Cannot open file for write: /build/core/dynamorioloc.cmake.tmpcf7e2
CMake Error: : System Error: Not a directory
CMake Error in core/CMakeLists.txt:
Could not open file for write in copy operation
/build/core/dynamorioloc.cmake
[...]
```
This is especially problematic when building in containers, as there is no `coredump` namespace in the Linux kernel and by that the settings can only be configured system-wide. Also tests with `ulimit -c 0` inside the container did not solve the issue.
I hope, documenting this as an issue helps others to find the root cause (which is absolutely non trivial) and just apply the mitigations.
Contributor guide
Assessment
This issue has not been assessed yet.