rr-debugger / rr-debugger/rr

rr crash on flush_syscallbuf

Open
#4,014 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

Hey, I found another crash around mprotect

[FATAL src/ReplaySession.cc:1562:flush_syscallbuf()] 
 (task 134826 (rec:134705) at time 3013)
 -> Assertion `t->stop_sig() == SIGSEGV && r.ip() == t->vm()->do_breakpoint_fault_addr()' failed to hold. Replay got unexpected signal (or none) 4 ip 0x792d9b18ba09 breakpoint_fault_addr 0x792d9b1cb787
Tail of trace dump:
{
  real_time:18714.582505 global_time:2993, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:134707, ticks:1691198
rax:0xffffffffffffffda rbx:0x792d8d5fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x189 rdi:0x792d9b1ba740 rbp:0x792d8c0bd760 rsp:0x792d8d5ffdf0 r8:0x0 r9:0x100792dffffffff r10:0x0 r11:0x246 r12:0x792d8c0bf640 r13:0x16 r14:0x792d8e2947d0 r15:0x7ffee8717900 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x792d8c0bf640 gs_base:0x0
}
{
  real_time:18714.582510 global_time:2994, event:`SYSCALLBUF_RESET' tid:134707, ticks:1691198
}
{
  real_time:18714.585313 global_time:2995, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:134705, ticks:1190144891
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x2000f000 rdi:0x0 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585349 global_time:2996, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:134705, ticks:1190144891
rax:0x792ca3ff1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x2000f000 rdi:0x0 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x792d9b057a80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x792ca3ff1000, length:0x2000f000, prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x2000f000 }
}
{
  real_time:18714.585381 global_time:2997, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:134705, ticks:1190144925
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x792d9b13d7b7 rsi:0xf000 rdi:0x792ca3ff1000 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585419 global_time:2998, event:`SYSCALL: munmap' (state:EXITING_SYSCALL) tid:134705, ticks:1190144925
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x792d9b13d7b7 rsi:0xf000 rdi:0x792ca3ff1000 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585475 global_time:2999, event:`SYSCALLBUF_FLUSH' tid:134705, ticks:1190145516
  { syscall:'openat', ret:0x7, size:0x10, desched:1 }
  { syscall:'readlinkat', ret:0xc, size:0x1c }
  { syscall:'fstatat', ret:0x0, size:0xa0 }
}
{
  real_time:18714.585481 global_time:3000, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:134705, ticks:1190145516
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffee8716e10 rsi:0x5401 rdi:0x7 rbp:0x59e7035c5cc0 rsp:0x681ffdd0 r8:0x7ffee8716e50 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffee8717060 r14:0x7ffee8717060 r15:0x792d9b057a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585486 global_time:3001, event:`SYSCALLBUF_RESET' tid:134705, ticks:1190145516
}
{
  real_time:18714.585514 global_time:3002, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:134705, ticks:1190145516
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffee8716e10 rsi:0x5401 rdi:0x7 rbp:0x59e7035c5cc0 rsp:0x681ffdd0 r8:0x7ffee8716e50 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffee8717060 r14:0x7ffee8717060 r15:0x792d9b057a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x792d9b057a80 gs_base:0x0
  { tid:134705, addr:0x7ffee8716e10, length:0x24 }
}
{
  real_time:18714.585613 global_time:3003, event:`SYSCALLBUF_FLUSH' tid:134705, ticks:1190148544
  { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
  { syscall:'close', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { start:0x792ca4000000, size:10000, prot:'r--' }
  { start:0x792ca4010000, size:4000, prot:'rw-' }
  { start:0x792ca4020000, size:4000, prot:'rw-' }
  { start:0x792ca4030000, size:4000, prot:'rw-' }
  { start:0x792ca4040000, size:4000, prot:'rw-' }
}
{
  real_time:18714.585619 global_time:3004, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:134705, ticks:1190148544
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x400f000 rdi:0x0 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585627 global_time:3005, event:`SYSCALLBUF_RESET' tid:134705, ticks:1190148544
}
{
  real_time:18714.585660 global_time:3006, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:134705, ticks:1190148544
rax:0x792c9fff1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x400f000 rdi:0x0 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x792d9b057a80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x792c9fff1000, length:0x400f000, prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x400f000 }
}
{
  real_time:18714.585692 global_time:3007, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:134705, ticks:1190148578
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x792d9b13d7b7 rsi:0xf000 rdi:0x792c9fff1000 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585722 global_time:3008, event:`SYSCALL: munmap' (state:EXITING_SYSCALL) tid:134705, ticks:1190148578
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x792d9b13d7b7 rsi:0xf000 rdi:0x792c9fff1000 rbp:0x7ffee8716fd0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x4022 r11:0x246 r12:0x7ffee87186c8 r13:0x59e6eb48fe50 r14:0x0 r15:0x792d9b21f040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585773 global_time:3009, event:`SYSCALLBUF_FLUSH' tid:134705, ticks:1190149137
  { syscall:'openat', ret:0x7, size:0x10, desched:1 }
  { syscall:'readlinkat', ret:0xc, size:0x1c }
  { syscall:'fstatat', ret:0x0, size:0xa0 }
}
{
  real_time:18714.585779 global_time:3010, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:134705, ticks:1190149137
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffee8716e10 rsi:0x5401 rdi:0x7 rbp:0x59e7035c5cc0 rsp:0x681ffdd0 r8:0x7ffee8716e50 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffee8717060 r14:0x7ffee8717060 r15:0x792d9b057a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x792d9b057a80 gs_base:0x0
}
{
  real_time:18714.585784 global_time:3011, event:`SYSCALLBUF_RESET' tid:134705, ticks:1190149137
}
{
  real_time:18714.585812 global_time:3012, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:134705, ticks:1190149137
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffee8716e10 rsi:0x5401 rdi:0x7 rbp:0x59e7035c5cc0 rsp:0x681ffdd0 r8:0x7ffee8716e50 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffee8717060 r14:0x7ffee8717060 r15:0x792d9b057a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x792d9b057a80 gs_base:0x0
  { tid:134705, addr:0x7ffee8716e10, length:0x24 }
}
{
  real_time:18714.590368 global_time:3013, event:`SYSCALLBUF_FLUSH' tid:134705, ticks:1191037696
  { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
  { syscall:'close', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'madvise', ret:0x0, size:0x10 }
  { syscall:'madvise', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'mprotect', ret:0x0, size:0x10 }
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'openat', ret:0x7, size:0x10, desched:1 }
  { syscall:'readlinkat', ret:0xc, size:0x1c }
  { syscall:'fstatat', ret:0x0, size:0xa0 }
  { start:0x792ca0000000, size:10000, prot:'r--' }
  { start:0x792ca0010000, size:4000, prot:'rw-' }
  { start:0x792ca0020000, size:4000, prot:'rw-' }
  { start:0x792ca0030000, size:4000, prot:'rw-' }
  { start:0x792ca0040000, size:4000, prot:'rw-' }
  { start:0x792ce4000000, size:10000, prot:'rw-' }
  { start:0x792ce4000000, size:10000, prot:'r--' }
  { start:0x77eb00000000, size:40000, prot:'rw-' }
  { start:0x77eb00080000, size:40000, prot:'rw-' }
  { start:0x77eb000c0000, size:40000, prot:'rw-' }
  { start:0x77eb00100000, size:40000, prot:'rw-' }
  { start:0x77eb00140000, size:40000, prot:'rw-' }
  { start:0x77eb00180000, size:40000, prot:'rw-' }
  { start:0x77eb00200000, size:40000, prot:'rw-' }
  { start:0x77eb00233000, size:d000, prot:'---' }
  { start:0x77eb00000000, size:40000, prot:'r--' }
  { start:0x77eb00080000, size:40000, prot:'r--' }
  { start:0x77eb000c0000, size:40000, prot:'r--' }
  { start:0x77eb00100000, size:40000, prot:'r--' }
  { start:0x77eb00140000, size:40000, prot:'r--' }
  { start:0x77eb00180000, size:40000, prot:'r--' }
  { start:0x77eb00200000, size:33000, prot:'r--' }
  { start:0x792cc4000000, size:10000, prot:'rw-' }
  { start:0x792cc4000000, size:10000, prot:'r--' }
  { start:0x77eb00040000, size:40000, prot:'rw-' }
  { start:0x77eb00740000, size:40000, prot:'rw-' }
  { start:0x77eb00780000, size:40000, prot:'rw-' }
  { start:0x77eb001c0000, size:40000, prot:'rw-' }
  { start:0x77eb00240000, size:500000, prot:'rw-' }
  { start:0x46a00100000, size:40000, prot:'rw-' }
  { start:0x46a00140000, size:40000, prot:'rw-' }
}
{
  real_time:18714.590378 global_time:3014, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:134705, ticks:1191037696
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffee8716dd0 rsi:0x5401 rdi:0x7 rbp:0x59e7035c5cc0 rsp:0x681ffdd0 r8:0x7ffee8716e10 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffee8717020 r14:0x7ffee8717020 r15:0x792d9b057a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x792d9b057a80 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x5b)[0x580f088a7fcb]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x152)[0x580f0877a572]
rr(+0xe30da)[0x580f0877b0da]
rr(+0xe32df)[0x580f0877b2df]
rr(+0xe3319)[0x580f0877b319]
rr(_ZN2rr13ReplaySession16flush_syscallbufEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x121e)[0x580f0881a3de]
rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x263)[0x580f0881c1d3]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x21d)[0x580f0881d98d]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xbf)[0x580f0883afef]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xf48)[0x580f0875ed48]
rr(_ZN2rr9GdbServer12serve_replayESt10shared_ptrINS_13ReplaySessionEERKNS0_6TargetEPVbNS_12DebuggerTypeERKNS0_15ConnectionFlagsE+0x977)[0x580f0875fb47]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x8f0)[0x580f08810820]
rr(main+0x1a1)[0x580f086edc71]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x737b27829d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x737b27829e40]
rr(_start+0x25)[0x580f086f0775]
=== End rr backtrace

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/ReplaySession.cc:1562 in flush_syscallbuf() and inspect the reported assertion alongside the trace tail around the mprotect and syscall-buffer events. Use the provided unexpected signal and breakpoint-fault addresses to investigate the replay failure. Done means the reported replay no longer crashes at this assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.