Replay divergence

The processes in questions do a bunch of complicated stuff, including user namespaces. However, an earlier version of this software recorded fine under rr (never tried this exact version though, so can't say if this is a regression). The value 0xffff000012345678 is something we pwrite to /proc/self/mem to test whether that mechanism is suitable for JITing, so that seems potentially related (though something more must be going on here, since we usually catch these writes).

[ERROR /home/keno/rr/src/Registers.cc:256:maybe_print_reg_mismatch()] rdx 0 != 0xffff000012345678 (replaying vs. recorded)
[FATAL /home/keno/rr/src/Registers.cc:366:compare_register_files()]
 (task 241681 (rec:241560) at time 10668)
 -> Assertion `!bail_error || match' failed to hold. Fatal register mismatch (ticks/rec:10900973/10900973)
Tail of trace dump:
{
  real_time:13731775.587633 global_time:10648, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10641061
rax:0xffffffffffffffda rbx:0x7f52a83dcd38 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dcd38 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.587661 global_time:10649, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10641061
rax:0x0 rbx:0x7f52a83dcd38 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dcd38 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.587789 global_time:10650, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10641630
rax:0xffffffffffffffda rbx:0x7f52a83dc278 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dc278 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.587818 global_time:10651, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10641630
rax:0x0 rbx:0x7f52a83dc278 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dc278 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.587954 global_time:10652, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10642189
rax:0xffffffffffffffda rbx:0x7f52a83dbdc0 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dbdc0 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.587983 global_time:10653, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10642189
rax:0x0 rbx:0x7f52a83dbdc0 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dbdc0 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588148 global_time:10654, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10642811
rax:0xffffffffffffffda rbx:0x7f52a83dbe98 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dbe98 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588177 global_time:10655, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10642811
rax:0x0 rbx:0x7f52a83dbe98 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dbe98 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588307 global_time:10656, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10643480
rax:0xffffffffffffffda rbx:0x7f52a83dd258 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dd258 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588336 global_time:10657, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10643480
rax:0x0 rbx:0x7f52a83dd258 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dd258 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa00 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa10 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588464 global_time:10658, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:241560, ticks:10644057
rax:0xffffffffffffffda rbx:0x7f52a83dc0d8 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dc0d8 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa10 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa20 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.588493 global_time:10659, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:241560, ticks:10644057
rax:0x0 rbx:0x7f52a83dc0d8 rcx:0xffffffffffffffff rdx:0x7fffffff rsi:0x81 rdi:0x7f52a83dc0d8 rbp:0x7f52a5482dd0 rsp:0x7ffe3cf4aa10 r8:0x0 r9:0x19550a0 r10:0x0 r11:0x246 r12:0x7ffe3cf4aa20 r13:0x7ffe3cf4b550 r14:0x7ffe3cf4b520 r15:0x0 rip:0x7f52a87a0310 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589378 global_time:10660, event:`SYSCALL: uname' (state:ENTERING_SYSCALL) tid:241560, ticks:10899954
rax:0xffffffffffffffda rbx:0x7ffe3cf4a7b0 rcx:0xffffffffffffffff rdx:0x1ac0d98 rsi:0x7f52a839d308 rdi:0x7ffe3cf4a7b0 rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a798 r8:0x1 r9:0x3 r10:0x803 r11:0x246 r12:0x1a80e10 r13:0x1ac00c0 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a849f4c7 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3f fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589419 global_time:10661, event:`SYSCALL: uname' (state:EXITING_SYSCALL) tid:241560, ticks:10899954
rax:0x0 rbx:0x7ffe3cf4a7b0 rcx:0xffffffffffffffff rdx:0x1ac0d98 rsi:0x7f52a839d308 rdi:0x7ffe3cf4a7b0 rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a798 r8:0x1 r9:0x3 r10:0x803 r11:0x246 r12:0x1a80e10 r13:0x1ac00c0 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a849f4c7 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3f fs_base:0x7f52a974ec80 gs_base:0x0
  { tid:241560, addr:0x7ffe3cf4a7b0, length:0x186 }
}
{
  real_time:13731775.589565 global_time:10662, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:241560, ticks:10900566
rax:0xffffffffffffffda rbx:0x181002 rcx:0xffffffffffffffff rdx:0x181002 rsi:0x7f52a8f80908 rdi:0xffffffffffffff9c rbp:0x7f52a8f80908 rsp:0x7ffe3cf4a720 r8:0x0 r9:0x2e r10:0x0 r11:0x246 r12:0x1a80e10 r13:0x1ac00c0 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a87a2641 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589633 global_time:10663, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:241560, ticks:10900566
rax:0xe rbx:0x181002 rcx:0xffffffffffffffff rdx:0x181002 rsi:0x7f52a8f80908 rdi:0xffffffffffffff9c rbp:0x7f52a8f80908 rsp:0x7ffe3cf4a720 r8:0x0 r9:0x2e r10:0x0 r11:0x246 r12:0x1a80e10 r13:0x1ac00c0 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a87a2641 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589761 global_time:10664, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:241560, ticks:10900575
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1000 rdi:0x0 rbp:0xffffffffffffffff rsp:0x7ffe3cf4a768 r8:0xffffffffffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x0 r14:0x1000 r15:0x22 rip:0x7f52a84cb093 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589829 global_time:10665, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:241560, ticks:10900575
rax:0x7f52a9755000 rbx:0x5 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1000 rdi:0x0 rbp:0xffffffffffffffff rsp:0x7ffe3cf4a768 r8:0xffffffffffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x0 r14:0x1000 r15:0x22 rip:0x7f52a84cb093 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7f52a974ec80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7f52a9755000, length:0x1000, prot_flags:"r-xp", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:13731775.589960 global_time:10666, event:`SYSCALL: pwrite64' (state:ENTERING_SYSCALL) tid:241560, ticks:10900760
rax:0xffffffffffffffda rbx:0x7f52a9755000 rcx:0xffffffffffffffff rdx:0x8 rsi:0x7ffe3cf4a7a8 rdi:0xe rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a790 r8:0xffffffffffffffff r9:0x0 r10:0x7f52a9755000 r11:0x246 r12:0xe r13:0x7f52a9261c70 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a87a27a3 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12 fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.589997 global_time:10667, event:`SYSCALL: pwrite64' (state:EXITING_SYSCALL) tid:241560, ticks:10900760
rax:0x8 rbx:0x7f52a9755000 rcx:0xffffffffffffffff rdx:0x8 rsi:0x7ffe3cf4a7a8 rdi:0xe rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a790 r8:0xffffffffffffffff r9:0x0 r10:0x7f52a9755000 r11:0x246 r12:0xe r13:0x7f52a9261c70 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a87a27a3 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x12 fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.590127 global_time:10668, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:241560, ticks:10900973
rax:0xffffffffffffffda rbx:0x7f52a9755000 rcx:0xffffffffffffffff rdx:0xffff000012345678 rsi:0x1000 rdi:0x7f52a9755000 rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a798 r8:0xffffffffffffffff r9:0x0 r10:0x574 r11:0x246 r12:0xe r13:0x7f52a9261c70 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a84cb157 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x7f52a974ec80 gs_base:0x0
}
{
  real_time:13731775.590169 global_time:10669, event:`SYSCALL: munmap' (state:EXITING_SYSCALL) tid:241560, ticks:10900973
rax:0x0 rbx:0x7f52a9755000 rcx:0xffffffffffffffff rdx:0xffff000012345678 rsi:0x1000 rdi:0x7f52a9755000 rbp:0x7ffe3cf4a960 rsp:0x7ffe3cf4a798 r8:0xffffffffffffffff r9:0x0 r10:0x574 r11:0x246 r12:0xe r13:0x7f52a9261c70 r14:0x1abf658 r15:0x1abe548 rip:0x7f52a84cb157 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x7f52a974ec80 gs_base:0x0
}
=== Start rr backtrace:
/home/keno/rr-build/bin/rr(_ZN2rr13dump_rr_stackEv+0x44)[0x557bfe88c4f9]
/home/keno/rr-build/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x1a2)[0x557bfe72809e]
/home/keno/rr-build/bin/rr(+0x2d3e4f)[0x557bfe74ce4f]
/home/keno/rr-build/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x62)[0x557bfe74d0aa]
/home/keno/rr-build/bin/rr(_ZN2rr9Registers22compare_register_filesEPNS_10ReplayTaskEPKcRKS0_S4_S6_NS_16MismatchBehaviorE+0x145)[0x557bfe7f48f7]
/home/keno/rr-build/bin/rr(_ZN2rr10ReplayTask13validate_regsEj+0x31a)[0x557bfe8145da]
/home/keno/rr-build/bin/rr(_ZN2rr13ReplaySession13enter_syscallEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x764)[0x557bfe7fe210]
/home/keno/rr-build/bin/rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0xf9)[0x557bfe801e25]
/home/keno/rr-build/bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x15b)[0x557bfe80320b]
/home/keno/rr-build/bin/rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0x10e)[0x557bfe81e8da]
/home/keno/rr-build/bin/rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x59)[0x557bfe726d5f]
/home/keno/rr-build/bin/rr(+0x38064d)[0x557bfe7f964d]
/home/keno/rr-build/bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3a3)[0x557bfe7fa01b]
/home/keno/rr-build/bin/rr(main+0x1f7)[0x557bfe8a3de3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fd7b0c801c1]
/home/keno/rr-build/bin/rr(_start+0x2a)[0x557bfe69fb1a]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:45073' /home/keno/.local/share/rr/julia-109/mmap_copy_724_julia

Start in src/Registers.cc at maybe_print_reg_mismatch() and compare_register_files(), then reproduce the recorded/replayed workload involving user namespaces and pwrite64 to /proc/self/mem. Done means identifying the cause of the rdx mismatch at global time 10668 and making replay match the recording for this scenario.