rr-debugger / rr-debugger/rr

JIT code seems to interfere with reverse execution.

Open
#3,461 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

First of all I just want to thank you all very much for creating this incredible tool. It has completely changed the way I debug and I'm very much in the "can't ever go back" camp. Big big thanks. Now onto the question...

I need to debug an appication that uses both OpenGL and Vulkan. Since RR does not appear to play with well with GPU graphics/compute, I run the application using Mesa's llvmpipe as a software based driver. This works well enough in that it makes RR usable, but often times when I am running a reverse-continue to the next breakpoint I end up stepping over some driver code and things all of a sudden get very slow. This slowness is accompanied by printouts like this:

Unable to find JITed code entry at address: 0x7f56abfffba0

By slow I mean, I recorded an application running for about 1 and half minutes which ended in a crash. I run rr replay -e to get to the end, and then I set a breakpoint at a method that I know to be in the call stack that caused the crash. This call would have occurred within seconds of the crash. I run reverse-continue. I start seeing printouts like above. I get about 50 of those before I see the message

warning: Temporarily disabling breakpoints for unloaded shared library <path to shared library that my breakpoint was in>

Maybe a few minutes later I see this message

Traceback (most recent call last):
  File "<string>", line 550, in render
  File "<string>", line 1178, in lines
gdb.error: Dwarf Error: Can't read DWARF data in section .eh_frame [in module <in-memory>]

And i'm back at the gdb prompt. The whole process took about 40 minutes. I never got to the breakpoint.

So, I'm just really not sure what is going on here, other than it seems like the JIT code that is being run by the software driver is messing things up. I have been able to successfully debug issues with rr using this software backend at times but its hit or miss. I'm not sure, but I recall sometimes being able to step over the JIT stuff and come out the other side, but many times it does seem to take things off the rails.

Any idea what might be going on here or tips for working around this? Most of the time I'm not interested in looking at any of the driver code, and would be happy if that could just be ignored by rr.

System info:
rr 5.6.0
gdb 12.1
linux kernel 6.2.1
CPU Intel i9-9900K x86_64
archlinux distro, up-to-date as of about 2 weeks ago.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names rr replay -e, reverse-continue, Mesa llvmpipe, and the GDB messages, but no repository files or tests. Start by reproducing the slowdown with the reported rr, GDB, kernel, and llvmpipe setup, then trace when JIT code and unloaded-library warnings appear. Done would be a confirmed cause or a documented workaround for reverse execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
devtools, reverse-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.