rr-debugger / rr-debugger/rr

switch_processes test failures because of LD_LIBRARY_PATH

Open
#3,606 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

This seems to be a "revival" of #2028. All other tests pass.

The "switch_processes" tests pass when LD_LIBRARY_PATH contains not more than 4 entries:

$ LD_LIBRARY_PATH="a:b:c:d" ctest -R 'switch_processes' -VV
UpdateCTestConfiguration  from :/tmp/rr/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/tmp/rr/build/DartConfiguration.tcl
Test project /tmp/rr/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1434
    Start 1434: switch_processes

1434: Test command: /usr/bin/bash "source_dir/src/test/switch_processes.run" "switch_processes" "" "bin_dir" "120"
1434: Test timeout computed to be: 10000000
1434: Test 'switch_processes' PASSED
1/2 Test #1434: switch_processes .................   Passed    1.83 sec
test 1435
    Start 1435: switch_processes-no-syscallbuf

1435: Test command: /usr/bin/bash "source_dir/src/test/switch_processes.run" "switch_processes" "-n" "bin_dir" "120"
1435: Test timeout computed to be: 10000000
1435: Test 'switch_processes' PASSED
2/2 Test #1435: switch_processes-no-syscallbuf ...   Passed    1.77 sec

The following tests passed:
        switch_processes
        switch_processes-no-syscallbuf

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   3.63 sec

and completely fails otherwise:

$ LD_LIBRARY_PATH="a:b:c:d:e" ctest -R 'switch_processes' -VV
UpdateCTestConfiguration  from :/tmp/rr/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/tmp/rr/build/DartConfiguration.tcl
Test project /tmp/rr/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1434
    Start 1434: switch_processes

1434: Test command: /usr/bin/bash "source_dir/src/test/switch_processes.run" "switch_processes" "" "bin_dir" "120"
1434: Test timeout computed to be: 10000000
1434: source_dir/src/test/util.sh: line 342: 3622140 Aborted                 (core dumped) _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT debug.err python3 $TESTDIR/$expectscript.py $RR_EXE $GLOBAL_OPTIONS replay -o-n -o-ix -o$TESTDIR/test_setup.gdb $replayargs
1434: Test 'switch_processes' FAILED: debug script failed
1434: --------------------------------------------------
1434: gdb_rr.log:
1434: 3622131: forking and exec'ing write_race-Oe0V77wjM...
1434: child 3622132
1434: GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9
1434: Copyright (C) 2021 Free Software Foundation, Inc.
1434: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
1434: This is free software: you are free to change and redistribute it.
1434: There is NO WARRANTY, to the extent permitted by law.
1434: Type "show copying" and "show warranty" for details.
1434: This GDB was configured as "x86_64-redhat-linux-gnu".
1434: Type "show configuration" for configuration details.
1434: For bug reporting instructions, please see:
1434: <https://www.gnu.org/software/gdb/bugs/>.
1434: Find the GDB manual and other documentation resources online at:
1434:     <http://www.gnu.org/software/gdb/documentation/>.
1434:
1434: For help, type "help".
1434: Type "apropos word" to search for commands related to "word"...
1434: /tmp/rr/src/test/test_setup.gdb:7: Error in sourced command file:
1434: No symbol table is loaded.  Use the "file" command.
1434: Reading symbols from /tmp/rr-test-switch_processes-Oe0V77wjM/target_process-Oe0V77wjM-0/mmap_clone_41_write_race-Oe0V77wjM...
1434: Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
1434: Really redefine built-in command "jump"? (y or n) [answered Y; input not from terminal]
1434: Remote debugging using 127.0.0.1:17663
1434:
1434: --------------------------------------------------
1434:  ---> Reached target process 3622132 at event 1001.
1434: --------------------------------------------------
1434: Reading symbols from /lib64/ld-linux-x86-64.so.2...
1434: Reading symbols from /tmp/rr/build/lib/rr/librrpreload.so...
1434: Reading symbols from /usr/lib64/libdl.so.2...
1434: Reading symbols from /usr/lib/debug/usr/lib64/libdl.so.2-2.34-60.el9.x86_64.debug...
1434: BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past end of file
1434: 0x00007fd7cfb75efb in mprotect () at ../sysdeps/unix/syscall-template.S:117
1434: 117       T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS,
1434: (rr) r 1
1434: y
1434: r 1
0x00007fd7cfb75f01      117     T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS,
1434: The program being debugged has been started already.
1434: Start it from the beginning? (y or n) y
Starting program: /tmp/rr-test-switch_processes-Oe0V77wjM/target_process-Oe0V77wjM-0/mmap_clone_41_write_race-Oe0V77wjM 1
1434: 3622131: forking and exec'ing write_race-Oe0V77wjM...
1434: child 3622132
1434:
1434: --------------------------------------------------
1434:  ---> Reached target process 3622132 at event 532.
1434: --------------------------------------------------
1434: warning: Build ID mismatch between current exec-file /tmp/rr-test-switch_processes-Oe0V77wjM/target_process-Oe0V77wjM-0/mmap_clone_41_write_race-Oe0V77wjM
1434: and automatically determined exec-file /tmp/rr-test-switch_processes-Oe0V77wjM/target_process-Oe0V77wjM-0/mmap_clone_4_target_process-Oe0V77wjM
1434: exec-file-mismatch handling is currently "ask"
1434: Load new symbol table from "/tmp/rr-test-switch_processes-Oe0V77wjM/target_process-Oe0V77wjM-0/mmap_clone_4_target_process-Oe0V77wjM"? (y or n) --------------------------------------------------
1434: debug.err:
1434: timeout 120 exceeded
1434: ====== /proc/3622141/status
1434: [...]
1434: Detaching from program: /tmp/rr/build/bin/rr, process 3622143
1434: [Inferior 1 (process 3622143) detached]
1434: --------------------------------------------------
1434: Test switch_processes failed, leaving behind /tmp/rr-test-switch_processes-Oe0V77wjM and /tmp/rr/build/rr-test-switch_processes-O6fkgPKb4
1434: To replay the failed test, run
1434:   _RR_TRACE_DIR=/tmp/rr-test-switch_processes-Oe0V77wjM rr replay
1/2 Test #1434: switch_processes .................***Failed  Error regular expression found in output. Regex=[FAILED]124.97 sec
test 1435
    Start 1435: switch_processes-no-syscallbuf

1435: Test command: /usr/bin/bash "source_dir/src/test/switch_processes.run" "switch_processes" "-n" "bin_dir" "120"
1435: Test timeout computed to be: 10000000
1435: source_dir/src/test/util.sh: line 342: 3622614 Aborted                 (core dumped) _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT debug.err python3 $TESTDIR/$expectscript.py $RR_EXE $GLOBAL_OPTIONS replay -o-n -o-ix -o$TESTDIR/test_setup.gdb $replayargs
1435: Test 'switch_processes' FAILED: debug script failed
1435: --------------------------------------------------
1435: gdb_rr.log:
1435: 3622601: forking and exec'ing write_race-F3qJErQSZ...
1435: child 3622602
1435: GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9
1435: Copyright (C) 2021 Free Software Foundation, Inc.
1435: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
1435: This is free software: you are free to change and redistribute it.
1435: There is NO WARRANTY, to the extent permitted by law.
1435: Type "show copying" and "show warranty" for details.
1435: This GDB was configured as "x86_64-redhat-linux-gnu".
1435: Type "show configuration" for configuration details.
1435: For bug reporting instructions, please see:
1435: <https://www.gnu.org/software/gdb/bugs/>.
1435: Find the GDB manual and other documentation resources online at:
1435:     <http://www.gnu.org/software/gdb/documentation/>.
1435:
1435: For help, type "help".
1435: Type "apropos word" to search for commands related to "word"...
1435: /tmp/rr/src/test/test_setup.gdb:7: Error in sourced command file:
1435: No symbol table is loaded.  Use the "file" command.
1435: Reading symbols from /tmp/rr-test-switch_processes-F3qJErQSZ/target_process-F3qJErQSZ-0/mmap_clone_39_write_race-F3qJErQSZ...
1435: Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
1435: Really redefine built-in command "jump"? (y or n) [answered Y; input not from terminal]
1435: Remote debugging using 127.0.0.1:18137
1435:
1435: --------------------------------------------------
1435:  ---> Reached target process 3622602 at event 1001.
1435: --------------------------------------------------
1435: Reading symbols from /tmp/rr/build/lib/rr/librrpreload.so...
1435: Reading symbols from /usr/lib64/libdl.so.2...
1435: Reading symbols from /usr/lib/debug/usr/lib64/libdl.so.2-2.34-60.el9.x86_64.debug...
1435: Reading symbols from /usr/lib64/libc.so.6...
1435: Reading symbols from /usr/lib/debug/usr/lib64/libc.so.6-2.34-60.el9.x86_64.debug...
1435: Reading symbols from /lib64/ld-linux-x86-64.so.2...
1435: BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past end of file
1435: 0x00007f06c5295efb in mprotect () at ../sysdeps/unix/syscall-template.S:117
1435: 117       T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS,
1435: (rr) r 1
1435: y
1435: r 1
0x00007f06c5295f01      117     T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS,
1435: The program being debugged has been started already.
1435: Start it from the beginning? (y or n) y
Starting program: /tmp/rr-test-switch_processes-F3qJErQSZ/target_process-F3qJErQSZ-0/mmap_clone_39_write_race-F3qJErQSZ 1
1435: 3622601: forking and exec'ing write_race-F3qJErQSZ...
1435: child 3622602
1435:
1435: --------------------------------------------------
1435:  ---> Reached target process 3622602 at event 518.
1435: --------------------------------------------------
1435: warning: Build ID mismatch between current exec-file /tmp/rr-test-switch_processes-F3qJErQSZ/target_process-F3qJErQSZ-0/mmap_clone_39_write_race-F3qJErQSZ
1435: and automatically determined exec-file /tmp/rr-test-switch_processes-F3qJErQSZ/target_process-F3qJErQSZ-0/mmap_clone_4_target_process-F3qJErQSZ
1435: exec-file-mismatch handling is currently "ask"
1435: Load new symbol table from "/tmp/rr-test-switch_processes-F3qJErQSZ/target_process-F3qJErQSZ-0/mmap_clone_4_target_process-F3qJErQSZ"? (y or n) --------------------------------------------------
1435: debug.err:
1435: timeout 120 exceeded
1435: ====== /proc/3622615/status
1435: [...]
1435: Detaching from program: /tmp/rr/build/bin/rr, process 3622617
1435: [Inferior 1 (process 3622617) detached]
1435: --------------------------------------------------
1435: Test switch_processes failed, leaving behind /tmp/rr-test-switch_processes-F3qJErQSZ and /tmp/rr/build/rr-test-switch_processes-Yha5Y4isb
1435: To replay the failed test, run
1435:   _RR_TRACE_DIR=/tmp/rr-test-switch_processes-F3qJErQSZ rr replay
2/2 Test #1435: switch_processes-no-syscallbuf ...***Failed  Error regular expression found in output. Regex=[FAILED]124.93 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) = 249.93 sec

The following tests FAILED:
        1434 - switch_processes (Failed)
        1435 - switch_processes-no-syscallbuf (Failed)
Errors while running CTest

Full log

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/test/switch_processes.run and src/test/util.sh, then reproduce both ctest switch_processes cases with four and five LD_LIBRARY_PATH entries. Compare the failing replay logs and test_setup.gdb behavior; done means both switch_processes tests pass with the longer path.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, cmake, cpp, linux, python
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.