JIT code seems to interfere with reverse execution.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.7k
- Forks
- 662
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 2
Description
First of all I just want to thank you all very much for creating this incredible tool. It has completely changed the way I debug and I'm very much in the "can't ever go back" camp. Big big thanks. Now onto the question...
I need to debug an appication that uses both OpenGL and Vulkan. Since RR does not appear to play with well with GPU graphics/compute, I run the application using Mesa's llvmpipe as a software based driver. This works well enough in that it makes RR usable, but often times when I am running a reverse-continue to the next breakpoint I end up stepping over some driver code and things all of a sudden get very slow. This slowness is accompanied by printouts like this:
Unable to find JITed code entry at address: 0x7f56abfffba0
By slow I mean, I recorded an application running for about 1 and half minutes which ended in a crash. I run rr replay -e to get to the end, and then I set a breakpoint at a method that I know to be in the call stack that caused the crash. This call would have occurred within seconds of the crash. I run reverse-continue. I start seeing printouts like above. I get about 50 of those before I see the message
warning: Temporarily disabling breakpoints for unloaded shared library <path to shared library that my breakpoint was in>
Maybe a few minutes later I see this message
Traceback (most recent call last):
File "<string>", line 550, in render
File "<string>", line 1178, in lines
gdb.error: Dwarf Error: Can't read DWARF data in section .eh_frame [in module <in-memory>]
And i'm back at the gdb prompt. The whole process took about 40 minutes. I never got to the breakpoint.
So, I'm just really not sure what is going on here, other than it seems like the JIT code that is being run by the software driver is messing things up. I have been able to successfully debug issues with rr using this software backend at times but its hit or miss. I'm not sure, but I recall sometimes being able to step over the JIT stuff and come out the other side, but many times it does seem to take things off the rails.
Any idea what might be going on here or tips for working around this? Most of the time I'm not interested in looking at any of the driver code, and would be happy if that could just be ignored by rr.
System info:
rr 5.6.0
gdb 12.1
linux kernel 6.2.1
CPU Intel i9-9900K x86_64
archlinux distro, up-to-date as of about 2 weeks ago.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report names rr replay -e, reverse-continue, Mesa llvmpipe, and the GDB messages, but no repository files or tests. Start by reproducing the slowdown with the reported rr, GDB, kernel, and llvmpipe setup, then trace when JIT code and unloaded-library warnings appear. Done would be a confirmed cause or a documented workaround for reverse execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- devtools, reverse-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100