rr-debugger / rr-debugger/rr

CI: Intermittent failure in vfork_read_clone_stress

Open
#3,129 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

https://buildkite.com/julialang/rr/builds/239#3c3e325a-988c-4285-b1ae-f5d878fef8ee



855/2761 Test #1158: vfork_read_clone_stress ...........................................***Failed  Error regular expression found in output. Regex=[FAILED] 15.64 sec
--
  | Targeting recorded pid 123298 ...
  | 127.0.0.1:41183: Connection timed out.
  | The program has no registers now.
  | source_dir/src/test/util.sh: line 296: 124257 Aborted                 _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT debug.err python3 $TESTDIR/$expectscript.py $RR_EXE $GLOBAL_OPTIONS replay -o-n -x $TESTDIR/test_setup.gdb $replayargs
  | Test 'vfork_read_clone_stress' FAILED: debug script failed
  | --------------------------------------------------
  | gdb_rr.log:
  | GNU gdb (Debian 10.1-2) 10.1.90.20210103-git
  | Copyright (C) 2021 Free Software Foundation, Inc.
  | License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  | This is free software: you are free to change and redistribute it.
  | There is NO WARRANTY, to the extent permitted by law.
  | Type "show copying" and "show warranty" for details.
  | This GDB was configured as "x86_64-linux-gnu".
  | Type "show configuration" for configuration details.
  | For bug reporting instructions, please see:
  | <https://www.gnu.org/software/gdb/bugs/>.
  | Find the GDB manual and other documentation resources online at:
  | <http://www.gnu.org/software/gdb/documentation/>.
  |  
  | For help, type "help".
  | Type "apropos word" to search for commands related to "word"...
  | Reading symbols from /cache/build/exclusive-amdci1-0/julialang/rr/obj/my_temp_parent_dir/jl_bBMnIl/rr-test-vfork_read_clone_stress-RTn2qpbrt/vfork_read_clone_stress-RTn2qpbrt-0/mmap_hardlink_4_vfork_read_clone_stress-RTn2qpbrt...
  | Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
  | 
  | --------------------------------------------------
  | ---> Reached target process 123298 at event 235.
  | --------------------------------------------------
  | Remote debugging using 127.0.0.1:58798
  | Reading symbols from /cache/build/exclusive-amdci1-0/julialang/rr/obj/lib/rr/librrpreload.so...
  | Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...
  | Reading symbols from /usr/lib/debug/.build-id/54/322f9a621b22e9ea3a4a96b6692f7a0f931b2d.debug...
  | Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
  | Reading symbols from /usr/lib/debug/.build-id/7d/74053de4b1acdbbc139fd54deaaa32b3b38655.debug...
  | Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...
  | Reading symbols from /usr/lib/debug/.build-id/23/b03cb94c34544ed3b222db64fe7b24cfd73fa1.debug...
  | Reading symbols from /lib64/ld-linux-x86-64.so.2...
  | Reading symbols from /usr/lib/debug/.build-id/b1/133999bbb1801a832010378ae667737cad3131.debug...
  | BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past end of file
  | vfork () at ../sysdeps/unix/sysv/linux/x86_64/vfork.S:41
  | 41	../sysdeps/unix/sysv/linux/x86_64/vfork.S: No such file or directory.
  | (rr) handle SIGKILL stop
  | c
  | handle SIGKILL stop
  | Signal        Stop	Print	Pass to program	Description
  | SIGKILL       Yes	Yes	Yes		Killed
  | (rr) c
  | Continuing.
  | rc
  | rc
  | child 123460
  | child 123450
  | child 123440
  | child 123433
  | child 123428
  | child 123414
  | child 123402
  | child 123391
  | child 123386
  | child 123379
  | child 123371
  | child 123365
  | child 123362
  | child 123356
  | child 123352
  | child 123298
  |  
  | Program received signal SIGKILL, Killed.
  | 0x0000000070000002 in syscall_traced ()
  | (rr) rc
  | Continuing.
  |  
  | Program stopped.
  | c
  | q
  | y
  | --------------------------------------------------
  | debug.err:
  | process 124334 sent SIGURG
  | ====== /proc/124263/status
  | Name:	python3
  | Umask:	0022
  | State:	S (sleeping)
  | Tgid:	124263
  | Ngid:	0
  | Pid:	124263
  | PPid:	124257
  | TracerPid:	0
  | Uid:	0	0	0	0
  | Gid:	0	0	0	0
  | FDSize:	64
  | Groups:	0
  | NStgid:	124263
  | NSpid:	124263
  | NSpgid:	124263
  | NSsid:	0
  | VmPeak:	   17032 kB
  | VmSize:	   17032 kB
  | VmLck:	       0 kB
  | VmPin:	       0 kB
  | VmHWM:	   10648 kB
  | VmRSS:	   10648 kB
  | RssAnon:	    4588 kB
  | RssFile:	    6060 kB
  | RssShmem:	       0 kB
  | VmData:	    4832 kB
  | VmStk:	     136 kB
  | VmExe:	    2624 kB
  | VmLib:	    2388 kB
  | VmPTE:	      76 kB
  | VmSwap:	       0 kB
  | HugetlbPages:	       0 kB
  | CoreDumping:	0
  | THP_enabled:	1
  | Threads:	1
  | SigQ:	15/2063165
  | SigPnd:	0000000000000000
  | ShdPnd:	0000000000000000
  | SigBlk:	0000000000000000
  | SigIgn:	0000000001001000
  | SigCgt:	0000000180000002
  | CapInh:	0000000000000000
  | CapPrm:	0000003fffffffff
  | CapEff:	0000003fffffffff
  | CapBnd:	0000003fffffffff
  | CapAmb:	0000000000000000
  | NoNewPrivs:	0
  | Seccomp:	0
  | Speculation_Store_Bypass:	thread vulnerable
  | Cpus_allowed:	ffffffff,ffffffff,ffffffff,ffffffff
  | Cpus_allowed_list:	0-127
  | Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
  | Mems_allowed_list:	0-1
  | voluntary_ctxt_switches:	82
  | nonvoluntary_ctxt_switches:	6
  | ====== /proc/124263/stack
  | ====== /proc/124316/status
  | Name:	gdb
  | State:	Z (zombie)
  | Tgid:	124316
  | Ngid:	0
  | Pid:	124316
  | PPid:	124263
  | TracerPid:	0
  | Uid:	0	0	0	0
  | Gid:	0	0	0	0
  | FDSize:	0
  | Groups:	0
  | NStgid:	124316
  | NSpid:	124316
  | NSpgid:	124316
  | NSsid:	124316
  | Threads:	1
  | SigQ:	15/2063165
  | SigPnd:	0000000000000000
  | ShdPnd:	0000000000020001
  | SigBlk:	0000000000000000
  | SigIgn:	0000000001001000
  | SigCgt:	0000000188034485
  | CapInh:	0000000000000000
  | CapPrm:	0000003fffffffff
  | CapEff:	0000003fffffffff
  | CapBnd:	0000003fffffffff
  | CapAmb:	0000000000000000
  | NoNewPrivs:	0
  | Seccomp:	0
  | Speculation_Store_Bypass:	thread vulnerable
  | Cpus_allowed:	ffffffff,ffffffff,ffffffff,ffffffff
  | Cpus_allowed_list:	0-127
  | Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
  | Mems_allowed_list:	0-1
  | voluntary_ctxt_switches:	161
  | nonvoluntary_ctxt_switches:	1038
  | ====== /proc/124316/stack
  | (file couldn't be opened)
  | ====== gdb -p 124334 -ex 'set confirm off' -ex 'set height 0' -ex 'thread apply all bt' -ex q </dev/null 2>&1
  | GNU gdb (Debian 10.1-2) 10.1.90.20210103-git
  | Copyright (C) 2021 Free Software Foundation, Inc.
  | License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  | This is free software: you are free to change and redistribute it.
  | There is NO WARRANTY, to the extent permitted by law.
  | Type "show copying" and "show warranty" for details.
  | This GDB was configured as "x86_64-linux-gnu".
  | Type "show configuration" for configuration details.
  | For bug reporting instructions, please see:
  | <https://www.gnu.org/software/gdb/bugs/>.
  | Find the GDB manual and other documentation resources online at:
  | <http://www.gnu.org/software/gdb/documentation/>.
  |  
  | For help, type "help".
  | Type "apropos word" to search for commands related to "word".
  | Attaching to process 124334
  | ptrace: No such process.
  | ====== gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:41183' -ex 'set confirm off' -ex 'set height 0' -ex 'info registers' -ex 'thread apply all bt' -ex q /cache/build/exclusive-amdci1-0/julialang/rr/obj/my_temp_parent_dir/jl_bBMnIl/rr-test-vfork_read_clone_stress-RTn2qpbrt/vfork_read_clone_stress-RTn2qpbrt-0/mmap_hardlink_4_vfork_read_clone_stress-RTn2qpbrt
  | </dev/null 2>&1
  | GNU gdb (Debian 10.1-2) 10.1.90.20210103-git
  | Copyright (C) 2021 Free Software Foundation, Inc.
  | License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  | This is free software: you are free to change and redistribute it.
  | There is NO WARRANTY, to the extent permitted by law.
  | Type "show copying" and "show warranty" for details.
  | This GDB was configured as "x86_64-linux-gnu".
  | Type "show configuration" for configuration details.
  | For bug reporting instructions, please see:
  | <https://www.gnu.org/software/gdb/bugs/>.
  | Find the GDB manual and other documentation resources online at:
  | <http://www.gnu.org/software/gdb/documentation/>.
  |  
  | For help, type "help".
  | Type "apropos word" to search for commands related to "word"...
  | Reading symbols from /cache/build/exclusive-amdci1-0/julialang/rr/obj/my_temp_parent_dir/jl_bBMnIl/rr-test-vfork_read_clone_stress-RTn2qpbrt/vfork_read_clone_stress-RTn2qpbrt-0/mmap_hardlink_4_vfork_read_clone_stress-RTn2qpbrt...
  | --------------------------------------------------
  | Test vfork_read_clone_stress failed, leaving behind /cache/build/exclusive-amdci1-0/julialang/rr/obj/my_temp_parent_dir/jl_bBMnIl/rr-test-vfork_read_clone_stress-RTn2qpbrt
  | To replay the failed test, run
  | _RR_TRACE_DIR=/cache/build/exclusive-amdci1-0/julialang/rr/obj/my_temp_parent_dir/jl_bBMnIl/rr-test-vfork_read_clone_stress-RTn2qpbrt rr replay

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Buildkite failure and the vfork_read_clone_stress test, then inspect source_dir/src/test/util.sh around line 296. Try the recorded replay command shown in the report and compare the GDB timeout and SIGKILL behavior. Done means identifying and fixing the intermittent failure so the stress test completes reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux, python
Domain
operating-systems, reverse-engineering, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
24/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.