When software breakpoints are managed by `lldb` (the client), the breakpoint stop PC is not adjusted, leading to problems
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Found while investigating https://github.com/llvm/llvm-project/pull/222009.
AArch64 reports a breakpoint PC as at the breakpoint, x86 and others report it afterwards. Normally lldb-server does this adjustment, but only if it knows about the breakpoint. If the server does not support the breakpoint packets, `lldb` will set software breaks itself, but it does not adjust the PC of breakpoint stop events so it gets confused.
I've never come across a server that doesn't support break packets but in theory it is valid, and there is code in lldb to handle it. You can hack lldb-server to look like one:
```
diff --git a/lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp b/lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp
index 38aeb1bb4090..c802ee9c549a 100644
--- a/lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp
+++ b/lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp
@@ -202,10 +202,6 @@ void GDBRemoteCommunicationServerLLGS::RegisterPacketHandlers() {
RegisterMemberFunctionHandler(
StringExtractorGDBRemote::eServerPacketType_x,
&GDBRemoteCommunicationServerLLGS::Handle_memory_read);
- RegisterMemberFunctionHandler(StringExtractorGDBRemote::eServerPacketType_Z,
- &GDBRemoteCommunicationServerLLGS::Handle_Z);
- RegisterMemberFunctionHandler(StringExtractorGDBRemote::eServerPacketType_z,
- &GDBRemoteCommunicationServerLLGS::Handle_z);
RegisterMemberFunctionHandler(
StringExtractorGDBRemote::eServerPacketType_QPassSignals,
&GDBRemoteCommunicationServerLLGS::Handle_QPassSignals);
@@ -225,9 +221,6 @@ void GDBRemoteCommunicationServerLLGS::RegisterPacketHandlers() {
RegisterMemberFunctionHandler(
StringExtractorGDBRemote::eServerPacketType_jLLDBTraceGetBinaryData,
&GDBRemoteCommunicationServerLLGS::Handle_jLLDBTraceGetBinaryData);
- RegisterMemberFunctionHandler(
- StringExtractorGDBRemote::eServerPacketType_jMultiBreakpoint,
- &GDBRemoteCommunicationServerLLGS::Handle_jMultiBreakpoint);
RegisterMemberFunctionHandler(
StringExtractorGDBRemote::eServerPacketType_jAcceleratorPluginInitialize,
&GDBRemoteCommunicationServerLLGS::Handle_jAcceleratorPluginInitialize);
```
In each case we start with:
```
$ ./bin/lldb-server gdbserver 127.0.0.1:1234 --log-channels "lldb break" -- /tmp/test.o
```
Then connect to the remote and try to run to main. `lldb-server` managed breakpoints work:
```
$ ./bin/lldb /tmp/test.o
(lldb) target create "/tmp/test.o"
Current executable set to '/tmp/test.o' (x86_64).
(lldb) gdb-remote 1234
Connection established.
Process 26190 stopped
* thread #1, name = 'test.o', stop reason = signal SIGSTOP
frame #0: 0x00007ffff7fe4cc0 ld-linux-x86-64.so.2`_start
(lldb) b main
Breakpoint 1: where = test.o`main + 8 at test.c:1:21, address = 0x0000555555555131
(lldb) c
1788946151.122799158 NativeProcessProtocol.cpp:SetSoftwareBreakpoint addr = 0x7ffff7fc4a00, size_hint = 1
1788946151.122860909 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint Overwriting bytes at 0x7ffff7fc4a00: 0xf3
1788946151.122910500 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint addr = 0x7ffff7fc4a00: SUCCESS
1788946151.122929573 NativeProcessProtocol.cpp:SetSoftwareBreakpoint addr = 0x555555555131, size_hint = 1
1788946151.122955799 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint Overwriting bytes at 0x555555555131: 0xb8
1788946151.122988701 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint addr = 0x555555555131: SUCCESS
Process 26190 resuming
(lldb) 1788946151.123524189 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 26190
1788946151.123564720 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded breakpoint size: 1
1788946151.123581886 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded pid 26190 tid 26190: changing PC from 0x7ffff7fc4a01 to 0x7ffff7fc4a00
1788946151.125494242 NativeProcessProtocol.cpp:RemoveSoftwareBreakpoint addr = 0x7ffff7fc4a00
1788946151.126093149 NativeProcessProtocol.cpp:SetSoftwareBreakpoint addr = 0x7ffff7fc4a00, size_hint = 1
1788946151.126109838 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint Overwriting bytes at 0x7ffff7fc4a00: 0xf3
1788946151.126123190 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint addr = 0x7ffff7fc4a00: SUCCESS
1788946151.126715422 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 26190
1788946151.126727343 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded breakpoint size: 1
1788946151.126733541 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded pid 26190 tid 26190: changing PC from 0x7ffff7fc4a01 to 0x7ffff7fc4a00
1788946151.203728437 NativeProcessProtocol.cpp:RemoveSoftwareBreakpoint addr = 0x7ffff7fc4a00
1788946151.204435825 NativeProcessProtocol.cpp:SetSoftwareBreakpoint addr = 0x7ffff7fc4a00, size_hint = 1
1788946151.204452753 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint Overwriting bytes at 0x7ffff7fc4a00: 0xf3
1788946151.204467773 NativeProcessProtocol.cpp:EnableSoftwareBreakpoint addr = 0x7ffff7fc4a00: SUCCESS
1788946151.204648733 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 26190
1788946151.204660654 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded breakpoint size: 1
1788946151.204667807 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded pid 26190 tid 26190: changing PC from 0x555555555132 to 0x555555555131
Process 26190 stopped
* thread #1, name = 'test.o', stop reason = breakpoint 1.1
frame #0: 0x0000555555555131 test.o`main at test.c:1:21
-> 1 int main() { return 0; }
(lldb) dis
test.o`main:
0x555555555129 <+0>: endbr64
0x55555555512d <+4>: pushq %rbp
0x55555555512e <+5>: movq %rsp, %rbp
-> 0x555555555131 <+8>: movl $0x0, %eax
0x555555555136 <+13>: popq %rbp
0x555555555137 <+14>: retq
```
Each breakpoint PC is adjusted back 1 byte and we silently continue past the dyld breakpoints.
With `lldb` managed breakpoints, this does not happen:
```
$ ./bin/lldb /tmp/test.o
(lldb) target create "/tmp/test.o"
Current executable set to '/tmp/test.o' (x86_64).
(lldb) gdb-remote 1234
Connection established.
Process 25768 stopped
* thread #1, name = 'test.o', stop reason = signal SIGSTOP
frame #0: 0x00007ffff7fe4cc0 ld-linux-x86-64.so.2`_start
(lldb) b main
Breakpoint 1: where = test.o`main + 8 at test.c:1:21, address = 0x0000555555555131
(lldb) c
Process 25768 resuming
(lldb) 1788945993.214264870 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 25768
1788945993.214283943 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded breakpoint size: 1
1788945993.214301109 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded pid 25768 no lldb software breakpoint found at current pc with adjustment: 0x7ffff7fc4a00
Process 25768 stopped
* thread #1, name = 'test.o', stop reason = signal SIGTRAP
frame #0: 0x00007ffff7fc4a01 ld-linux-x86-64.so.2`_dl_debug_state at dl-debug_state.c:28:1
(lldb) c
Process 25768 resuming
(lldb) 1788945997.074505329 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 25768
1788945997.074541569 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded breakpoint size: 1
1788945997.074564934 NativeProcessProtocol.cpp:FixupBreakpointPCAsNeeded pid 25768 no lldb software breakpoint found at current pc with adjustment: 0x7ffff7fc4a00
Process 25768 stopped
* thread #1, name = 'test.o', stop reason = signal SIGTRAP
frame #0: 0x00007ffff7fc4a01 ld-linux-x86-64.so.2`_dl_debug_state at dl-debug_state.c:28:1
(lldb) c
Process 25768 resuming
(lldb) 1788946000.381864548 NativeProcessLinux.cpp:MonitorBreakpoint received breakpoint event, pid = 25768
Process 25768 stopped
* thread #1, name = 'test.o', stop reason = signal SIGTRAP
frame #0: 0x0000555555555132 test.o`main at test.c:1:21
-> 1 int main() { return 0; }
(lldb) breakpoint list
Current breakpoints:
1: name = 'main', locations = 1, resolved = 1, hit count = 0
1.1: where = test.o`main + 8 at test.c:1:21, address = 0x0000555555555131, resolved, hit count = 0
(lldb) breakpoint list -i
Current breakpoints:
Kind: shared-library-event
-1: names = {'_dl_debug_state', 'rtld_db_dlactivity', '__dl_rtld_db_dlactivity', 'r_debug_state', '_r_debug_state', '_rtld_debug_state'}, language = c, module = ld-linux-x86-64.so.2, locations = 1, resolved = 1,
hit count = 0
-1.1: where = ld-linux-x86-64.so.2`_dl_debug_state at dl-debug_state.c:28:1, address = 0x00007ffff7fc4a00, resolved, hit count = 0
(lldb) dis
test.o`main:
0x555555555129 <+0>: endbr64
0x55555555512d <+4>: pushq %rbp
0x55555555512e <+5>: movq %rsp, %rbp
0x555555555131 <+8>: movl $0x0, %eax
0x555555555136 <+13>: popq %rbp
0x555555555137 <+14>: retq
```
To get to main I have to continue a few times, and beause the PC is off by one it isn't shown in disassembly.
As far as I know this would effect x86, SystemZ and any flavour of Windows (see `GetSoftwareBreakpointPCOffset`).
I thought of having `lldb-server` look back to see if the previous instruction is a software break. However this is dangerous if your software break encoding could be the end of a valid instruction. For example `0xCC` might appear in `0xAABBCC`, and have generated a trap for other reasons.
The solution is probably to have the client apply the change if it knows that it is the one managing software breakpoints.
Contributor guide
Research direction
Start with NativeProcessProtocol::FixupBreakpointPCAsNeeded and NativeProcessLinux::MonitorBreakpoint, then trace how client-managed software breakpoints are recorded and how remote stop events are handled. Reproduce with lldb-server after disabling its breakpoint packet handlers, and verify that client-managed breakpoint stops report the breakpoint address while unrelated SIGTRAP events remain unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100