DynamoRIO / DynamoRIO/dynamorio

CRASH ffmpeg on AArch64 crashes on dr_app_setup_and_start()

Open
#7,717 7 comments 0 reactions 0 assignees View on GitHub
OpSys-AArch64
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 15h
Merged PRs (30d)
31

Description

**Describe the bug**
When attaching drmemtrace to ffmpeg using `dr_app_setup_and_start()`, DR crashes with this error message:

```console
# benchmarks/video_transcode_bench/ffmpeg -y -i Netflix_Boat_4096x2160_60fps_10bit_420.y4m -crf 23 -preset 6 -g 80 -threads 1 -c:v libsvtav1 -f ivf -svtav1-params lp=1 /tmp/ffmpeg-svt_M6_NETFLIX_ElFuente_1920x1080to1280x720_lanc_14296frames_2997fps_003724_003802_Q23c.bin
[..]

[1] 902470 segmentation fault (core dumped) benchmarks/video_transcode_bench/ffmpeg -y -i -crf 23 -preset 6 -g 80 1 -c:
```

**To Reproduce**
1. Define variables needed for building ffmpeg with DynamoRIO. The build script assumes that the DR_ROOT is a tarball from https://github.com/DynamoRIO/dynamorio/releases
``` console
# # Ensure that $DR_ROOT/tools/lib64/debug/;$DR_ROOT/lib64/debug/;$DR_ROOT/ext/lib64/debug/;$DR_ROOT/tools/lib64/debug/libdrmemtrace.so are valid paths, adjust if necessary.
# export DR_ROOT=/home/smahar/DynamoRIO-AArch64-Linux-11.90.20395/
# export DYNAMORIO_OPTIONS="-client_lib $DR_ROOT/tools/lib64/debug/libdrmemtrace.so -loglevel 3;;-offline -code_api -loglevel 3"
# export LD_LIBRARY_PATH="$DR_ROOT/tools/lib64/debug/;$DR_ROOT/lib64/debug/;$DR_ROOT/ext/lib64/debug/;[DCPerf root directory]/benchmarks/video_transcode_bench/ffmpeg_build/lib64/"
```
2. A modified ‘n8.0’ version of ffmpeg with support for DynamoRIO’s `dr_app_setup_and_start()` and scripts to build it is available here: https://github.com/suyashmahar/DCPerf . To build this version, run `git clone https://github.com/suyashmahar/DCPerf`. and follow the steps here: https://github.com/suyashmahar/DCPerf/blob/export-D86680182-to-v2-beta/packages/video_transcode_bench/README.md .
> [!NOTE]
> In case the ffmpeg build fails, please check if these paths are valid: `$DR_ROOT/tools/lib64/debug/`, `$DR_ROOT/lib64/debug/`, and `$DR_ROOT/ext/lib64/debug/`. Once verified, clean the existing build with `rm -rf benchmarks benchmark_installs.txt` and try to build it again.
3. Once built, grab the input netflix sequence using: `wget http://download.opencontent.netflix.com.s3.amazonaws.com/ElFuente/Netflix_Boat_4096x2160_60fps_10bit_420.y4m`
4. Run ffmpeg: `benchmarks/video_transcode_bench/ffmpeg -y -i Netflix_Boat_4096x2160_60fps_10bit_420.y4m -crf 23 -preset 6 -g 80 -threads 1 -c:v libsvtav1 -f ivf -svtav1-params lp=1 /tmp/ffmpeg-svt_M6_NETFLIX_ElFuente_1920x1080to1280x720_lanc_14296frames_2997fps_003724_003802_Q23c.bin`

Please also answer these questions:
- What happens when you run without any client?
Crashes with a SIGSEGV, same behavior
- What happens when you run with debug build ("-debug" flag to drrun/drconfig/drinject)?
Not sure how to do this. Passing `-debug` before or after `;;` in `DYNAMORIO_OPTIONS` results in an error message saying unknown flag `-debug`. I’m already the debug version of the shared objects if that is more relevant for internal attach/detach:
```console
# ldd benchmarks/video_transcode_bench/ffmpeg | grep dy
libdynamorio.so => /home/smahar/DynamoRIO-AArch64-Linux-11.90.20395//lib64/debug/libdynamorio.so (0x0000000071000000)
```

Please provide a symbolized callstack of any crash or assert so we do not have to speculate or guess as to where it is occurring (see our [Linux gdb instructions](https://dynamorio.org/page_debugging.html#autotoc_md146) and [Windows windbg instructions](https://dynamorio.org/page_debugging.html#autotoc_md157) to see how to load DynamoRIO symbols for a callstack).

Running ffmpeg under gdb:
```
[After continuing SIGILL messages…]
(gdb) c
Continuing.

Thread 17 "vf#0:0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffbe850f100 (LWP 1746086)]
(gdb) bt
#0 0x0000000071453fb0 in get_thread_private_dcontext () at /home/runner/work/dynamorio/dynamorio/core/unix/os.c:3177
#1 0x000000007147aa34 in main_signal_handler_C (sig=4, siginfo=0xfffbe850d320, ucxt=0xfffbe850d3a0, xsp=0xfffbe850d320 "\004") at /home/runner/work/dynamorio/dynamorio/core/unix/signal.c:5712
#2 0x000000007144f7a0 in _dynamorio_runtime_resolve () at /home/runner/work/dynamorio/dynamorio/core/arch/aarch64/aarch64.asm:534
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

```

**Expected behavior**
FFMPEG should not crash.

**Versions**
- What version of DynamoRIO are you using? DynamoRIO-AArch64-Linux-11.90.20395
- Does the latest build from https://github.com/DynamoRIO/dynamorio/releases solve the problem? This is the latest version.
- What operating system version are you running on? ("Windows 10" is *not* sufficient: give the release number.) CentOS 9 on aarch64. Linux hostname 6.13.2-custom_string #1 SMP Tue Sep 2 10:38:48 PDT 2025 aarch64 aarch64 aarch64 GNU/Linux
- Is your application 32-bit or 64-bit? 64-bit aarch64

**Additional context**

The test suite for attach detach is passing for this version of DR:
```console
# ctest -j 5 -R client.attach
Test project /home/smahar/dynamorio/build
Start 138: code_api|client.attach_test
Start 140: code_api|client.attach_blocking
Start 141: code_api|client.attach_state
Start 143: code_api|client.attach_memory_dump_test
Start 359: code_api|client.attach-memory-dump-syscall-test
1/5 Test #143: code_api|client.attach_memory_dump_test ........... Passed 0.21 sec
2/5 Test #138: code_api|client.attach_test ....................... Passed 0.32 sec
3/5 Test #140: code_api|client.attach_blocking ................... Passed 0.32 sec
4/5 Test #359: code_api|client.attach-memory-dump-syscall-test ... Passed 0.32 sec
5/5 Test #141: code_api|client.attach_state ...................... Passed 5.01 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 5.04 sec

# ctest -j 5 -R client.detach
Test project /home/smahar/dynamorio/build
Start 139: code_api|client.detach_test
1/1 Test #139: code_api|client.detach_test ...... Passed 0.22 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.24 sec
```

[1] Patch for enabling DynamoRIO in ffmpeg:

```diff
diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
index de607cac93..7b3468f882 100644
--- a/fftools/ffmpeg.c
+++ b/fftools/ffmpeg.c
@@ -23,6 +23,24 @@
* multimedia converter based on the FFmpeg libraries
*/

+// DynamoRIO required start
+#define LINUX
+#if defined(__x86_64__) || defined(__i386__)
+#define X86_64
+#elif defined(__aarch64__) || defined(__arm__)
+#define ARM_64
+#endif
+
+#include "dr_api.h"
+
+#define TRACE_DURATION_S_DEFAULT 2
+#define TRACE_DURATION_S_ENV "TRACE_DURATION_S"
+
+#define ITERATION_WAIT_S_DEFAULT 10
+#define ITERATION_WAIT_S_ENV "ITERATION_WAIT_S"
+
+// DynamoRIO required end
+
#include "config.h"

#include
@@ -960,10 +978,60 @@ static int64_t getmaxrss(void)
#endif
}

+
+void *tracing_thread(void *arg);
+
+void *tracing_thread(void* arg) {
+ // Do nothing for now
+
+ int trace_duration_s = TRACE_DURATION_S_DEFAULT;
+ int iteration_wait_s = ITERATION_WAIT_S_DEFAULT;
+
+ char *trace_dur_raw = getenv(TRACE_DURATION_S_ENV);
+ if (trace_dur_raw) {
+ errno = 0;
+ trace_duration_s = strtol(trace_dur_raw, NULL, 0);
+ if (errno != 0) {
+ perror("Unable to parse TRACE_DURATION_S env var");
+ exit(1);
+ }
+ }
+
+ char *iter_wait_raw = getenv(ITERATION_WAIT_S_ENV);
+ if (iter_wait_raw) {
+ errno = 0;
+ iteration_wait_s = strtol(iter_wait_raw, NULL, 0);
+ if (errno != 0) {
+ perror("Unable to parse ITERATION_WAIT_S env var");
+ exit(1);
+ }
+ }
+
+ printf("Tracing duration set to %d, wait between iteration set to %d.\n",
+ trace_duration_s, iteration_wait_s);
+
+ sleep(1);
+ int trace_iter = 0;
+ while (1) {
+ dr_app_setup_and_start();
+ printf("Sleeping for %d seconds\n", trace_duration_s);
+ sleep(trace_duration_s);
+ dr_app_stop_and_cleanup();
+
+ printf("Completed %d iteration\n", trace_iter);
+ trace_iter++;
+ sleep(iteration_wait_s);
+ }
+}
+
+
int main(int argc, char **argv)
{
Scheduler *sch = NULL;

+ pthread_t tracing_thread_handler;
+ pthread_create(&tracing_thread_handler, NULL, &tracing_thread, NULL);
+
int ret;
BenchmarkTimeStamps ti;

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.