Breakpointing/JIT

I've encountered the following issue. Suppose we set a breakpoint in JITted code. We will look at the value that is there, write the breakpoint byte and continue. Later, we come back and restore the byte that we read. However, in the mean time the JIT could have come along and modified the code, causing the written back value to no longer be the value that was actually supposed to be there. Of course this is also an issue for plain old gdb, but it's somewhat more problematic for rr, because in reverse mode, it can easily happen that rr goes back to before the jitted code was there, but still wants to set a breakpoint (vs. regular debugging where such an operation makes little sense since we don't know what code will be there a priori).

Methods that I can think of by which JITs allocate modify executable memory:

1. /proc/self/mem/process_vm_writev.
2. Dual mapping with one RX and one RW page.
3. RWX pages

For 1., I think a simple check in the relevant code path to update any breakpoint would be sufficient. 2&3 are more tricky. We'd probably have to detect this situation in the address space code and do the appropriate dance to disable writing/executing in lock step (i.e. when executing writing is disabled, but we catch that trap, flip the permissions, and do the same thing vice versa once we get back to executing). This could be sped up by using hardware breakpoints, though of course those are a limited resource.

Thoughts?

No file, test, or entry point is named. Start by tracing breakpoint handling and the address-space paths that modify executable memory, then compare the /proc/self/mem, process_vm_writev, dual-mapping, and RWX cases described here. Done means breakpoint bytes remain correct when JIT code changes, including reverse execution before the code was allocated.