rr-debugger / rr-debugger/rr

Issues with low ulimits during record

Open
#2,521 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Description

We have a couple of issues when ulimits are low:

keno@arctic4:~/rr-clean-build$ /bin/sh -c 'ulimit -n 100; ./bin/rr record ls'
rr: Saving execution to trace directory `/home/keno/.local/share/rr/ls-79'.
[FATAL /home/keno/rr-clean/src/Task.cc:2950:spawn() errno: ECHILD] Tracee died before reaching SIGSTOP
Child's message: error duping to RR_MAGIC_SAVE_DATA_FD (EBADF)
=== Start rr backtrace:
./bin/rr(_ZN2rr13dump_rr_stackEv+0x5d)[0x55fd42a343ab]
./bin/rr(_ZN2rr15notifying_abortEv+0x57)[0x55fd42a34349]
./bin/rr(_ZN2rr12FatalOstreamD1Ev+0x34)[0x55fd428bfbc0]
./bin/rr(_ZN2rr4Task5spawnERNS_7SessionERNS_8ScopedFdEPS3_PiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorISC_SaISC_EESJ_i+0x7dd)[0x55fd429ff10b]
./bin/rr(_ZN2rr13RecordSessionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorIS6_SaIS6_EESD_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEiNS_7BindCPUES8_PKNS_9TraceUuidEb+0x32a)[0x55fd42909546]
./bin/rr(_ZN2rr13RecordSession6createERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EESB_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEhNS_7BindCPUERKS7_PKNS_9TraceUuidEb+0x9f8)[0x55fd42908e0c]
./bin/rr(+0x37d468)[0x55fd428fb468]
./bin/rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3da)[0x55fd428fc1e0]
./bin/rr(main+0x21f)[0x55fd42a4f388]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f39286370b3]
./bin/rr(_start+0x2e)[0x55fd427f85be]
=== End rr backtrace
Aborted
keno@arctic4:~/rr-clean-build$ ./bin/rr record /bin/sh -c 'ulimit -n 100; ls'
rr: Saving execution to trace directory `/home/keno/.local/share/rr/sh-7'.
/home/keno/rr-clean/src/preload/syscallbuf.c:475: Fatal error: "Failed to dup desched fd"
Aborted

I'm thinking for the first, we just ignore the error if we can't dup to RR_MAGIC_SAVE_DATA_FD, but of course we'll have the same problem with the socket fd also, though luckily we can already dynamically assign that, so it's probably just a matter of looking at the ulimit and picking an fd below it. For the desched fd, the two options I see are to just keep the tmp fd we already have if the dup fails, or alternatively having the tracer also dynamically assign that fd (and informing the syscallbuf library what it should be) to stay below the ulimit. I'm also wondering if we perhaps want to try raising the ulimit by two to accommodate our extra fds, but emulating the original ulimit (e.g. if issues only occur in the program being traced at certain ulimits, it would be nice if the extra fds we need for rr wouldn't affect that behavior).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the low-ulimit cases with the commands in the issue. Inspect Task.cc around spawn() and src/preload/syscallbuf.c around the desched-fd failure, then trace handling of RR_MAGIC_SAVE_DATA_FD and the socket fd. Done means rr record no longer aborts under the reported limit while preserving the traced program's effective ulimit.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.