rr-debugger / rr-debugger/rr

Instruction stream corrupted after futex system call

Open
#2,842 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

When recording my application with rr, it crashes consistently with a corrupt stack trace. After carefully stepping through the instructions up to the moment of the crash, it seems as though executing a syscall instruction in funlockfile corrupts the instructions at the site of the system call, crashing the next call to funlockfile.

This is the complete assembly for the function in glibc (2.31):

00000000000661e0 <_IO_funlockfile@@GLIBC_2.2.5>:
   661e0:       f3 0f 1e fa             endbr64 
   661e4:       48 8b bf 88 00 00 00    mov    0x88(%rdi),%rdi
   661eb:       8b 47 04                mov    0x4(%rdi),%eax
   661ee:       83 e8 01                sub    $0x1,%eax
   661f1:       89 47 04                mov    %eax,0x4(%rdi)
   661f4:       75 35                   jne    6622b <_IO_funlockfile@@GLIBC_2.2.5+0x4b>
   661f6:       48 c7 47 08 00 00 00    movq   $0x0,0x8(%rdi)
   661fd:       00 
   661fe:       64 8b 14 25 18 00 00    mov    %fs:0x18,%edx
   66205:       00 
   66206:       85 d2                   test   %edx,%edx
   66208:       75 06                   jne    66210 <_IO_funlockfile@@GLIBC_2.2.5+0x30>
   6620a:       83 2f 01                subl   $0x1,(%rdi)
   6620d:       c3                      retq   
   6620e:       66 90                   xchg   %ax,%ax
   66210:       87 07                   xchg   %eax,(%rdi)
   66212:       83 f8 01                cmp    $0x1,%eax
   66215:       7e 14                   jle    6622b <_IO_funlockfile@@GLIBC_2.2.5+0x4b>
   66217:       45 31 d2                xor    %r10d,%r10d
   6621a:       ba 01 00 00 00          mov    $0x1,%edx
   6621f:       be 81 00 00 00          mov    $0x81,%esi
   66224:       b8 ca 00 00 00          mov    $0xca,%eax
   66229:       0f 05                   syscall 
   6622b:       c3                      retq   
   6622c:       0f 1f 40 00             nopl   0x0(%rax)

This is what I see in gdb (Ubuntu 9.2-0ubuntu1~20.04):

# Instruction pointer is at 0x7fe08258f229, the address of the syscall.
# Examine the next instructions before we execute the syscall.
(rr) x/b 0x7fe08258f229
0x7fe08258f229 <__funlockfile+73>:      0x0f
0x7fe08258f22a <__funlockfile+74>:      0x05
0x7fe08258f22b <__funlockfile+75>:      0xc3
0x7fe08258f22c: 0x0f
0x7fe08258f22d: 0x1f
0x7fe08258f22e: 0x40
0x7fe08258f22f: 0x00

These match the instructions of the assembly in glibc. Now, stepi over the syscall and re-examine the same addresses:

(rr) si
(rr) x/b 0x7fe08258f229
0x7fe08258f229 <__funlockfile+73>:      0xe9
0x7fe08258f22a <__funlockfile+74>:      0xeb
0x7fe08258f22b <__funlockfile+75>:      0xf4
0x7fe08258f22c: 0x18
0x7fe08258f22d: 0x00
0x7fe08258f22e: 0x90
0x7fe08258f22f: 0x90

What's happened!? The instructions at those addresses have been changed. The next call into this function will crash. The new instructions look like a jump to an address that holds a move instruction:

   0x7fe08258f224 <__funlockfile+68>       mov    $0xca,%eax                                                                                                                       
   0x7fe08258f229 <__funlockfile+73>       jmpq   0x7fe08271e719                                                                                                                   
   0x7fe08258f22e                             nop                                                                                                                                 │
   0x7fe08258f22f                              nop 
(rr) x/i 0x7fe08271e719
   0x7fe08271e719:      mov    %rsp,0x70001010

Any help or advice here would be greatly appreciated.

Versions in play here:
gcc : Ubuntu 9.3.0-17ubuntu1 20.04
gdb : Ubuntu 9.2-0ubuntu1 20.04
rr : 5.4.0
glibc : Ubuntu GLIBC 2.31-0ubuntu9.2
kernel: 5.8.0-48-generic #54~20.04.1-Ubuntu SMP x86_64
cpu: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the recording around glibc's funlockfile entry point and the futex syscall, then inspect the instruction bytes before and after stepping over the syscall. The issue provides no rr source file or test to target; done means identifying why the syscall site changes and preventing the resulting instruction corruption, with a regression test if the relevant test location can be found.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.