TestBreakOnLambdaCapture.py is flakey on Windows on Arm
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
~~It is failing 9/10 times on our downstream lldb bot:~~
This test fails occasionally on our buildbot, which uses lldb-server. The test stalls 9/10 times when not using lldb-server.
```
FAIL: lldb-api :: functionalities/breakpoint/breakpoint_on_lambda_capture/TestBreakOnLambdaCapture.py (360 of 2968)
******************** TEST 'lldb-api :: functionalities/breakpoint/breakpoint_on_lambda_capture/TestBreakOnLambdaCapture.py' FAILED ********************
Script:
--
C:/ProgramData/scoop/apps/python/current/python.exe C:/buildbot/workspace/lldb-aarch64-windows/llvm-project/lldb\test\API\dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=C:/buildbot/workspace/lldb-aarch64-windows/build/./lib --env LLVM_INCLUDE_DIR=C:/buildbot/workspace/lldb-aarch64-windows/build/include --env LLVM_TOOLS_DIR=C:/buildbot/workspace/lldb-aarch64-windows/build/./bin --triple aarch64-pc-windows-msvc --build-dir C:/buildbot/workspace/lldb-aarch64-windows/build/lldb-test-build --lldb-module-cache-dir C:/buildbot/workspace/lldb-aarch64-windows/build/lldb-test-build/module-cache-lldb\lldb-api --clang-module-cache-dir C:/buildbot/workspace/lldb-aarch64-windows/build/lldb-test-build/module-cache-clang\lldb-api --executable C:/buildbot/workspace/lldb-aarch64-windows/build/./bin/lldb.exe --lldb-python-dir C:\buildbot\workspace\lldb-aarch64-windows\build\Lib\site-packages --compiler C:/buildbot/workspace/lldb-aarch64-windows/build/./bin/clang.exe --dsymutil C:/buildbot/workspace/lldb-aarch64-windows/build/./bin/dsymutil.exe --make C:/ProgramData/scoop/shims/make.exe --llvm-tools-dir C:/buildbot/workspace/lldb-aarch64-windows/build/./bin --lldb-obj-root C:/buildbot/workspace/lldb-aarch64-windows/build/tools/lldb --lldb-libs-dir C:/buildbot/workspace/lldb-aarch64-windows/build/./lib --cmake-build-type Release --skip-category=watchpoint --env LLDB_LAUNCH_FLAG_USE_PIPES=1 C:\buildbot\workspace\lldb-aarch64-windows\llvm-project\lldb\test\API\functionalities\breakpoint\breakpoint_on_lambda_capture -p TestBreakOnLambdaCapture.py --timeout 600
--
Exit Code: 1
Command Output (stdout):
--
Skipping the following test categories: watchpoint, libc++, libstdcxx, dwo, dsym, gmodules, debugserver, objc, fork, pexpect
--
Command Output (stderr):
--
FAIL: LLDB (C:\buildbot\workspace\lldb-aarch64-windows\build\bin\clang.exe-aarch64) :: test_break_on_lambda_capture (TestBreakOnLambdaCapture.TestBreakOnLambdaCapture.test_break_on_lambda_capture)
Log Files:
- C:\buildbot\workspace\lldb-aarch64-windows\build\lldb-test-build\functionalities\breakpoint\breakpoint_on_lambda_capture\TestBreakOnLambdaCapture\Failure_test_break_on_lambda_capture.log
======================================================================
FAIL: test_break_on_lambda_capture (TestBreakOnLambdaCapture.TestBreakOnLambdaCapture.test_break_on_lambda_capture)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\buildbot\workspace\lldb-aarch64-windows\llvm-project\lldb\test\API\functionalities\breakpoint\breakpoint_on_lambda_capture\TestBreakOnLambdaCapture.py", line 54, in test_break_on_lambda_capture
self.assertStopReason(
File "C:\buildbot\workspace\lldb-aarch64-windows\llvm-project\lldb\packages\Python\lldbsuite\test\lldbtest.py", line 3104, in assertStopReason
self.fail(self._formatMessage(msg, error))
AssertionError: trace (2) != breakpoint (3)
Config=aarch64-C:\buildbot\workspace\lldb-aarch64-windows\build\bin\clang.exe
----------------------------------------------------------------------
Ran 1 test in 1.485s
FAILED (failures=1)
```
When run on its own, it stalls after a few cycles of continuing from the breakpoint.
These logs are from a local, non-lldb-server run:
```
Looking at thread: thread #1: tid = 0x2e4c, 0x00007ff6673f11e0 a.out` `public: void __cdecl Foo::usleep_helper(this=0x000000f25675f9f0)'::`1'::::operator()(void) const at main.cpp:11:7, stop reason = breakpoint 2.1 --- 3
Looking at thread: thread #2: tid = 0x1778, 0x00007ff6673f11e4 a.out` `public: void __cdecl Foo::usleep_helper(this=0x000000f256affe00)'::`1'::::operator()(void) const at main.cpp:11:7 --- 1
Looking at thread: thread #3: tid = 0x0848, 0x00007ffb3db32ea4 ntdll.dll`NtWaitForWorkViaWorkerFactory + 4 --- 1
2
Process::ResumeSynchronous -- locking run lock
Process::PrivateResume() m_stop_id = 26, public state: stopped private state: stopped
Thread::PushPlan(0x000001C6215D7060): "Single stepping past breakpoint site 2 at 0x7ff6673f11e0", tid = 0x2e4c.
lldb_private::ThreadPlan::WillResume Thread #1 (0x000001C6215D7060): tid = 0x2e4c, pc = 0x7ff6673f11e0, sp = 0xf25675f9b0, fp = 0xf25675fa50, plan = 'Step over breakpoint trap', state = stepping, stop others = 1
Process::DisableSoftwareBreakpoint (breakID = 2) addr = 0x7ff6673f11e0
Process::DisableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0 -- SUCCESS
lldb_private::ThreadPlan::WillResume Thread #2 (0x000001C621BBFFB0): tid = 0x1778, pc = 0x7ff6673f11e4, sp = 0xf256affdc0, fp = 0xf256affeb0, plan = 'base plan', state = suspended, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #3 (0x000001C621EDCDD0): tid = 0x0848, pc = 0x7ffb3db32ea4, sp = 0xf256bffc30, fp = 0xf256bffeb0, plan = 'base plan', state = suspended, stop others = 0
Process::PrivateResume WillResume direction=0
(plugin = windows, state = running)
Process::PrivateResume thinks the process has resumed.
timeout =
Current Plan for thread 1(000001C6215D7060) (0x2e4c, stepping): Step over breakpoint trap being asked whether we should report run.
timeout = , event_sp)...
Current Plan for thread 2(000001C621BBFFB0) (0x1778, suspended): base plan being asked whether we should report run.
Current Plan for thread 3(000001C621EDCDD0) (0x0848, suspended): base plan being asked whether we should report run.
Process::ShouldBroadcastEvent (000001C6260FAE60) => new state: running, last broadcast state: running - YES
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) broadcasting new state running (old state running) to hijacked
Process::lldb_private::Process::PushProcessIOHandler pushing IO handler
Process::lldb_private::Process::HandlePrivateEvent updated m_iohandler_sync to 10
timeout = , event_sp)...
(plugin = windows, state = running, restarted = 0)
timeout = , event_sp) => running
timeout = , event_sp)...
(plugin = windows, state = stopped)
(plugin = windows, state = stopped, stop_id = 27
ThreadList::lldb_private::ThreadList::ShouldStop: 3 threads, 2 unsuspended threads
Thread::lldb_private::Thread::ShouldStop for tid = 0x2e4c 0x2e4c, pc = 0x00007ff6673f11e0, should_stop = 0 (ignore since no stop reason)
Thread::lldb_private::Thread::ShouldStop for tid = 0x1778 0x1778, should_stop = 0 (ignore since thread was suspended)
ThreadList::lldb_private::ThreadList::ShouldStop overall should_stop = 0
ThreadList::lldb_private::ThreadList::ShouldReportStop 3 threads
Thread::ShouldReportStop() tid = 0x2e4c: returning vote 0 (thread didn't stop for a reason.)
Thread::ShouldReportStop() tid = 0x1778: returning vote 0 (temporary state was suspended or invalid)
Thread::ShouldReportStop() tid = 0x0848: returning vote 0 (temporary state was suspended or invalid)
Returning no opinion
Process::ShouldBroadcastEvent: should_resume: 1 state: stopped was_restarted: 0 report_stop_vote: 0.
Process::ShouldBroadcastEvent (000001C6260FA6E0) Restarting process from state: stopped
Process::PrivateResume() m_stop_id = 27, public state: running private state: stopped
lldb_private::ThreadPlan::WillResume Thread #1 (0x000001C6215D7060): tid = 0x2e4c, pc = 0x7ff6673f11e0, sp = 0xf25675f9b0, fp = 0xf25675fa50, plan = 'Step over breakpoint trap', state = stepping, stop others = 1
lldb_private::ThreadPlan::WillResume Thread #2 (0x000001C621BBFFB0): tid = 0x1778, pc = 0x7ff6673f11e0, sp = 0xf256affdc0, fp = 0xf256affeb0, plan = 'base plan', state = suspended, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #3 (0x000001C621EDCDD0): tid = 0x0848, pc = 0x7ffb3db32ea4, sp = 0xf256bffc30, fp = 0xf256bffeb0, plan = 'base plan', state = suspended, stop others = 0
Process::PrivateResume WillResume direction=0
(plugin = windows, state = running)
Process::PrivateResume thinks the process has resumed.
Process::ShouldBroadcastEvent (000001C6260FA6E0) => new state: stopped, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state stopped (old state running): should_broadcast == false
timeout = , event_sp)...
Process::ShouldBroadcastEvent (000001C6260FA820) => new state: running, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state running (old state running): should_broadcast == false
timeout = , event_sp)...
(plugin = windows, state = stopped)
(plugin = windows, state = stopped, stop_id = 28
ThreadList::lldb_private::ThreadList::ShouldStop: 3 threads, 2 unsuspended threads
Thread::lldb_private::Thread::ShouldStop(000001C6215D7060) for tid = 0x2e4c 0x2e4c, pc = 0x00007ff6673f11e4
^^^^^^^^ Thread::ShouldStop Begin ^^^^^^^^
Plan stack initial state:
thread #1: tid = 0x2e4c:
Active plan stack:
Element 0: Base thread plan.
Element 1: Single stepping past breakpoint site 2 at 0x7ff6673f11e0
Step over breakpoint stopped for reason: trace.
Plan Step over breakpoint trap should stop: 0.
Completed step over breakpoint plan.
Process::EnableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0
Process::EnableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0 -- SUCCESS
Plan Step over breakpoint trap auto-continue: true.
Popping plan: "Step over breakpoint trap", tid = 0x2e4c.
Plan stack final state:
thread #1: tid = 0x2e4c:
Active plan stack:
Element 0: Base thread plan.
Completed plan stack:
Element 0: Single stepping past breakpoint site 2 at 0x7ff6673f11e0
vvvvvvvv Thread::ShouldStop End (returning 0) vvvvvvvv
Thread::lldb_private::Thread::ShouldStop for tid = 0x1778 0x1778, should_stop = 0 (ignore since thread was suspended)
ThreadList::lldb_private::ThreadList::ShouldStop overall should_stop = 0
ThreadList::lldb_private::ThreadList::ShouldReportStop 3 threads
Thread::ShouldReportStop() tid = 0x2e4c: returning vote for complete stack's back plan
Returning vote: no
Thread::ShouldReportStop() tid = 0x1778: returning vote 0 (temporary state was suspended or invalid)
Thread::ShouldReportStop() tid = 0x0848: returning vote 0 (temporary state was suspended or invalid)
Returning no
Process::ShouldBroadcastEvent: should_resume: 1 state: stopped was_restarted: 0 report_stop_vote: -1.
Process::ShouldBroadcastEvent (000001C6260FA640) Restarting process from state: stopped
Process::PrivateResume() m_stop_id = 28, public state: running private state: stopped
Thread::PushPlan(0x000001C621BBFFB0): "Single stepping past breakpoint site 2 at 0x7ff6673f11e0", tid = 0x1778.
lldb_private::ThreadPlan::WillResume Thread #1 (0x000001C6215D7060): tid = 0x2e4c, pc = 0x7ff6673f11e4, sp = 0xf25675f9b0, fp = 0xf25675fa50, plan = 'base plan', state = suspended, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #2 (0x000001C621BBFFB0): tid = 0x1778, pc = 0x7ff6673f11e0, sp = 0xf256affdc0, fp = 0xf256affeb0, plan = 'Step over breakpoint trap', state = stepping, stop others = 1
Process::DisableSoftwareBreakpoint (breakID = 2) addr = 0x7ff6673f11e0
Process::DisableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0 -- SUCCESS
lldb_private::ThreadPlan::WillResume Thread #3 (0x000001C621EDCDD0): tid = 0x0848, pc = 0x7ffb3db32ea4, sp = 0xf256bffc30, fp = 0xf256bffeb0, plan = 'base plan', state = suspended, stop others = 0
Process::PrivateResume WillResume direction=0
(plugin = windows, state = running)
Process::PrivateResume thinks the process has resumed.
Process::ShouldBroadcastEvent (000001C6260FA640) => new state: stopped, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state stopped (old state running): should_broadcast == false
timeout = , event_sp)...
Process::ShouldBroadcastEvent (000001C6260FAC80) => new state: running, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state running (old state running): should_broadcast == false
timeout = , event_sp)...
(plugin = windows, state = stopped)
(plugin = windows, state = stopped, stop_id = 29
ThreadList::lldb_private::ThreadList::ShouldStop: 3 threads, 1 unsuspended threads
Thread::lldb_private::Thread::ShouldStop(000001C621BBFFB0) for tid = 0x1778 0x1778, pc = 0x00007ff6673f11e4
^^^^^^^^ Thread::ShouldStop Begin ^^^^^^^^
Plan stack initial state:
thread #2: tid = 0x1778:
Active plan stack:
Element 0: Base thread plan.
Element 1: Single stepping past breakpoint site 2 at 0x7ff6673f11e0
Step over breakpoint stopped for reason: trace.
Plan Step over breakpoint trap should stop: 0.
Completed step over breakpoint plan.
Process::EnableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0
Process::EnableSoftwareBreakpoint (site_id = 2) addr = 0x7ff6673f11e0 -- SUCCESS
Plan Step over breakpoint trap auto-continue: true.
Popping plan: "Step over breakpoint trap", tid = 0x1778.
Plan stack final state:
thread #2: tid = 0x1778:
Active plan stack:
Element 0: Base thread plan.
Completed plan stack:
Element 0: Single stepping past breakpoint site 2 at 0x7ff6673f11e0
vvvvvvvv Thread::ShouldStop End (returning 0) vvvvvvvv
ThreadList::lldb_private::ThreadList::ShouldStop overall should_stop = 0
ThreadList::lldb_private::ThreadList::ShouldReportStop 3 threads
Thread::ShouldReportStop() tid = 0x2e4c: returning vote 0 (temporary state was suspended or invalid)
Thread::ShouldReportStop() tid = 0x1778: returning vote for complete stack's back plan
Returning vote: no
Thread::ShouldReportStop() tid = 0x0848: returning vote 0 (temporary state was suspended or invalid)
Returning no
Process::ShouldBroadcastEvent: should_resume: 1 state: stopped was_restarted: 0 report_stop_vote: -1.
Process::ShouldBroadcastEvent (000001C6260FA6E0) Restarting process from state: stopped
Process::PrivateResume() m_stop_id = 29, public state: running private state: stopped
lldb_private::ThreadPlan::WillResume Thread #1 (0x000001C6215D7060): tid = 0x2e4c, pc = 0x7ff6673f11e4, sp = 0xf25675f9b0, fp = 0xf25675fa50, plan = 'base plan', state = running, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #2 (0x000001C621BBFFB0): tid = 0x1778, pc = 0x7ff6673f11e4, sp = 0xf256affdc0, fp = 0xf256affeb0, plan = 'base plan', state = running, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #3 (0x000001C621EDCDD0): tid = 0x0848, pc = 0x7ffb3db32ea4, sp = 0xf256bffc30, fp = 0xf256bffeb0, plan = 'base plan', state = running, stop others = 0
Process::PrivateResume WillResume direction=0
(plugin = windows, state = running)
Process::PrivateResume thinks the process has resumed.
Process::ShouldBroadcastEvent (000001C6260FA6E0) => new state: stopped, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state stopped (old state running): should_broadcast == false
timeout = , event_sp)...
Process::ShouldBroadcastEvent (000001C6260FAD20) => new state: running, last broadcast state: running - NO
Process::lldb_private::Process::HandlePrivateEvent (pid = 7340) suppressing state running (old state running): should_broadcast == false
timeout = , event_sp)...
```
I think it may be something to do with stepping off of the breakpoint that thread 1 has hit, while thread 2 is at the same PC but has not hit the breakpoint:
```
lldb_private::ThreadPlan::WillResume Thread #1 (0x000001C6215D7060): tid = 0x2e4c, pc = 0x7ff6673f11e0, sp = 0xf25675f9b0, fp = 0xf25675fa50, plan = 'Step over breakpoint trap', state = stepping, stop others = 1
lldb_private::ThreadPlan::WillResume Thread #2 (0x000001C621BBFFB0): tid = 0x1778, pc = 0x7ff6673f11e0, sp = 0xf256affdc0, fp = 0xf256affeb0, plan = 'base plan', state = suspended, stop others = 0
lldb_private::ThreadPlan::WillResume Thread #3 (0x000001C621EDCDD0): tid = 0x0848, pc = 0x7ffb3db32ea4, sp = 0xf256bffc30, fp = 0xf256bffeb0, plan = 'base plan', state = suspended, stop others = 0
```
The PC is 0x7ff6673f11e0 for the first 2 threads.
Making the test program's loops finite does not fix the stall. So I think something is stuck within lldb.
This test is very similar to TestTwoHitsOneActual.py, which is also flakey on Windows on Arm. However when I run that test locally, it passes all the time.
Contributor guide
Research direction
Start with lldb/test/API/functionalities/breakpoint/breakpoint_on_lambda_capture/TestBreakOnLambdaCapture.py, especially the assertion at line 54, and run it on the aarch64-pc-windows-msvc configuration with and without lldb-server. Review the failure log and breakpoint continuation behavior; done means the test no longer flakes or stalls on the reported Windows Arm configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100