rr-debugger / rr-debugger/rr

Assertion failure when waiting on subprocess.

Open
#2,398 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

An assertion failure takes place on a debian VM running on i7-8550U when replaying a trace of code that is seemingly waiting on a subprocess to finish.
The relevant output from rr replay -

[FATAL /home/roc/rr/rr/src/Task.cc:2239:write_bytes_helper()]
 (task 11576 (rec:28769) at time 94583)
 -> Assertion `nwritten == buf_size' failed to hold. Should have written 2368 bytes to 0x7f8b7ccf9738, but only wrote 2248
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x28)[0x55faa8]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x385)[0x49c9e5]
rr(_ZN2rr21EmergencyDebugOstreamD2Ev+0xd6)[0x4a8656]
rr(_ZN2rr4Task18write_bytes_helperENS_10remote_ptrIvEElPKvPbj+0x395)[0x547615]
rr(_ZN2rr10ReplayTask19set_data_from_traceEv+0x83)[0x51f963]
rr(_ZN2rr13ReplaySession23emulate_signal_deliveryEPNS_10ReplayTaskEi+0xd8)[0x50bdc8]
rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x131)[0x510e21]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0xe9)[0x511a19]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xc3)[0x527543]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xe20)[0x49b100]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x4c0)[0x49c2c0]
rr[0x50750e]
rr(_ZN2rr13ReplayCommand3runERSt6vectorISsSaISsEE+0x7cb)[0x508d3b]
rr(main+0x244)[0x452504]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f22ac69b2e1]
rr[0x4529ef]
=== End rr backtrace

Backtrace of rr from GDB

#0  0x0000000070000002 in ?? ()
#1  0x00007f8bf3f2b5ba in _raw_syscall () at /home/roc/rr/rr/src/preload/raw_syscall.S:120
#2  0x00007f8bf3f27cd5 in traced_raw_syscall (call=call@entry=0x7f8b7ccf9fa0) at /home/roc/rr/rr/src/preload/syscallbuf.c:222
#3  0x00007f8bf3f2a389 in syscall_hook_internal (call=0x7f8b7ccf9fa0) at /home/roc/rr/rr/src/preload/syscallbuf.c:2653
#4  syscall_hook (call=0x7f8b7ccf9fa0) at /home/roc/rr/rr/src/preload/syscallbuf.c:2687
#5  0x00007f8bf3f27b3a in _syscall_hook_trampoline () at /home/roc/rr/rr/src/preload/syscall_hook.S:282
#6  0x00007f8bf3f27b6a in __morestack () at /home/roc/rr/rr/src/preload/syscall_hook.S:417
#7  0x00007f8bf3f27b88 in _syscall_hook_trampoline_48_3d_00_f0_ff_ff () at /home/roc/rr/rr/src/preload/syscall_hook.S:428
#8  0x00007f8bf31b0147 in __waitid (idtype=P_PID, id=28773, infop=0x7f8b90bf6608, options=16777220) at ../sysdeps/unix/sysv/linux/waitid.c:29
#9  0x000056358cedb02d in shared_child::sys::sys::wait_without_reaping::h572512d81fc9a2d5 (handle=...) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/shared_child-0.3.4/src/sys/unix.rs:19
#10 0x000056358ced9ca7 in shared_child::SharedChild::wait::hbacca39975165a30 (self=0x7f8b74001808) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/shared_child-0.3.4/src/lib.rs:146
#11 0x000056358ce99ab0 in duct::ChildHandle::wait::h0c7b4987065e7e92 (self=0x7f8b74001808, mode=duct::WaitMode::Blocking) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:1162
#12 0x000056358ce97b0a in duct::HandleInner::wait::h1ec1575ebc092812 (self=0x7f8b74001800, mode=duct::WaitMode::Blocking) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:1103
#13 0x000056358ce97bd1 in duct::HandleInner::wait::h1ec1575ebc092812 (self=0x7f8b90bf7380, mode=duct::WaitMode::Blocking) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:1108
#14 0x000056358ce95b46 in duct::Handle::wait::h0cf81bb2e796cc89 (self=0x7f8b90bf7380) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:987
#15 0x000056358ce96b8a in duct::Handle::output::h1c201b396d3a2140 (self=...) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:1051
#16 0x000056358ce94bba in duct::Expression::run::hd4a5adf37e06182e (self=0x7f8b90bf7970) at /home/e/.cargo/registry/src/github.com-1ecc6299db9ec823/duct-0.12.0/src/lib.rs:272
#17 0x000056358ca65c68 in talpid_core::ping_monitor::imp::internal_ping::h73ac6a194424f703 (ip=..., timeout_secs=2000, interface=..., exit_on_first_reply=true) at talpid-core/src/ping_monitor/unix.rs:49
#18 0x000056358ca65bb0 in talpid_core::ping_monitor::imp::ping::h6c109e5440063362 (ip=..., timeout_secs=2000, interface=...) at talpid-core/src/ping_monitor/unix.rs:40
#19 0x000056358c945427 in talpid_core::tunnel::wireguard::WireguardTunnel::start::_$u7b$$u7b$closure$u7d$$u7d$::h940b91200810c3cd () at talpid-core/src/tunnel/wireguard/mod.rs:211
#20 0x000056358ca94623 in std::sys_common::backtrace::__rust_begin_short_backtrace::h6830289098e3adee (f=...) at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/sys_common/backtrace.rs:127
#21 0x000056358ca45306 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h38128cbbe6913176 ()
    at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/thread/mod.rs:470
#22 0x000056358c9487a6 in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h50e33431d8ef77cb (self=..., _args=())
    at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/panic.rs:315
#23 0x000056358caaa97e in std::panicking::try::do_call::hf5f330ca0b702647 (data=0x7f8b90bf81f0 "") at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/panicking.rs:296
#24 0x000056358da7a64a in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:80
#25 0x000056358caa9160 in std::panicking::try::h0312c923bb3aafbd (f=...) at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/panicking.rs:275
#26 0x000056358c948b16 in std::panic::catch_unwind::h8cdec8d8359e01b3 (f=...) at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/panic.rs:394
#27 0x000056358ca4498d in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hd940cc09039793ad () at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libstd/thread/mod.rs:469
#28 0x000056358c97c494 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h2367c41653d298bc () at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/libcore/ops/function.rs:227
#29 0x000056358da5f7ff in _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hcc647de57eb732d8 ()
    at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/liballoc/boxed.rs:922
#30 0x000056358da799c0 in _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h4fb327f2066a5953 ()
    at /rustc/96d07e0ac9f0c56b95a2561c6cedac0b23a5d2a3/src/liballoc/boxed.rs:922
#31 std::sys_common::thread::start_thread::h267750d1d78a965d () at src/libstd/sys_common/thread.rs:13
#32 std::sys::unix::thread::Thread::new::thread_start::h645c01e371e9ae8e () at src/libstd/sys/unix/thread.rs:79
#33 0x00007f8bf36b54a4 in start_thread (arg=0x7f8b90bfa700) at pthread_create.c:456
#34 0x00007f8bf31e0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Could the differences between Ubuntu and Debian be the cause of this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rr/src/Task.cc at write_bytes_helper(), then follow ReplayTask::set_data_from_trace() and ReplaySession::emulate_signal_delivery() from the reported backtrace. Reproduce rr replay on the Debian VM with the trace and compare behavior on Ubuntu. Done means identifying the cause of the short write and preventing the assertion failure, with replay still working.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux, rust
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.