DynamoRIO / DynamoRIO/dynamorio
Trace barriers of 32-bit Linux sysenter cause many cache exits and repeated-forever ibl misses
- Dominant language
- C
- Stars
- 3.2k
- Forks
- 629
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 30
Description
32-bit api.startstop sometimes hangs on Linux.
Often, it churns up the CPU: attaching gdb I see the main thread at the
write syscall for the "all done" print, before it has set should_spin to
false. The other threads are all running the IBL looking up their
sideline_func. sleep(0) can be a nop for some libc implementations: is that it?
But, even with sched_yield instead of sleep, they spin. 3 or 4 are in sched_yield at any one time
but the others are all doing dispatch operations. The main thread really never gets scheduled enough over a 90s time period to make it past the other side of its print to set the condition variable?!? The 10 other threads are able to starve it?
The loop is like this:
```
dispatch: target = 0x0804924b
Entry into F4(0x0804924b).0x4f3bd040 (shared)
Exit from F21(0x08048b80).0x4f3cd0e9 (shared)
(block ends with syscall)
Entry into do_syscall to execute a non-ignorable system call
system call 158
Exit from system call
post syscall: sysnum=0x0000009e, result=0x00000000 (0)
dispatch: target = 0xf77a3bee
Entry into F11(0xf77a3bee).0x4f3bd0ec (shared)
Exit from sourceless ibl: bb ret
(target 0xf76129ac in cache but not lookup table)
dispatch: target = 0xf76129ac
Entry into F2(0xf76129ac).0x4f3bd018 (shared)
Exit from sourceless ibl: bb ret
(target 0x0804924b in cache but not lookup table)
```
So we have several problems:
1) The syscall is not inlined
2) The ibl targets are never put into the table
Is this restricted to 32-bit b/c of vsyscall making it statically harder to
know the syscall #? Can these 10 threads going through dispatch really
starve out the main thread for 90s?
To solve #1 some options are:
A) Treat "call *gs:0x10" as the syscall like we do for WOW64
B) Convert "call *gs:0x10" to a direct call via -indcall2direct and
inline its entry -- though will confuse clients. Today
-code_api does this:
options->indcall2direct = false;
options->inline_ignored_syscalls = false;
C) Whenever we see "mov $imm,eax; call *gs:0x10", if $imm is ignorable,
target a special bb that duplicates the vsyscall entry+sysenter but is
linkable?
However, for any of those, we have the problem of the sysenter control flow
gap -- will we mark the other side as a trace head so its ret will find its
target? We certainly made sure that happened on Windows in the past,
though I'm not sure about today on Linux with -code_api. With
-disable_traces the ret target should be found.
For #2 we need to know why it's not being added:
Exit from sourceless ibl: bb ret
(target 0xf76359ac in cache but not lookup table)
fragment_add_ibl_target tag 0xf76359ac, branch 0, F0
dispatch: target = 0xf76359ac
Entry into F2(0xf76359ac).0x47909018 (shared)
So it fails to find the fragment for the target. It's b/c it only looks
for a trace. It's the old issue of a dispatch-syscall permanently
interrupting a trace chain: we're supposed to mark the post-syscall point
as a trace head b/c the pre-syscall is a trace.
But with -disable_traces we still get the hang even though the loop is tighter:
```
Exit from F10(0xf77b6be0).0x4a4cd181 (shared)
(block ends with syscall)
Entry into do_syscall to execute a non-ignorable system call
system call 158
Exit from system call
post syscall: sysnum=0x0000009e, result=0x00000000 (0)
finished handling system call
dispatch: target = 0xf77b6bee
Entry into F11(0xf77b6bee).0x4a4cd19c (shared)
```
The bottom line is that exiting to dispatch in a loop is enough to cause a
major perf problem.
For this test we could switch to a pthread cond var or a futex.
Contributor guide
Assessment
This issue has not been assessed yet.