llvm / llvm/llvm-project

[lldb] Unreal Engine editor launch under lldb wedges during init on Windows (target threads self-Wait, no game tick)

Open
#211,170 1 comment 0 reactions 0 assignees View on GitHub
hang lldb platform:windows
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

## Environment
- lldb / lldb-dap **22.1.8**, revision `ca7933e47d3a3451d81e72ac174dcb5aa28b59d1`.
- **Build is patched** (a lazy PDB/SymbolFile-creation memory patch; see `upstream/draft-pr-lazy-pdbindex.md` in the repro repo below — an upstream PR is being prepared) and built `-DLLDB_ENABLE_PYTHON=OFF -DLLDB_ENABLE_LUA=OFF`. See the
"control not run" note below — this matters for attribution.
- Windows 11 (x86_64). Target: Unreal Engine 5.8 editor with engine debug symbols installed
(1134 PDBs). Observed on a 32 GB machine and on a 64 GB machine.

## Summary
Launching the UE editor under lldb-dap with `symbols.load-on-demand true` and a source breakpoint,
the target loads ~862 modules and then, at **t ≈ 80–95 s (immediately after `SessionServices.dll`
loads)**, all target threads go to `Wait` with CPU ≈ 0 and the process never reaches its game tick
loop. It stays wedged indefinitely (no tick at >600 s; reproduced across 4 runs). The lldb debug
loop itself is healthy and idle — the target has deadlocked, not lldb.

## Steps to reproduce
1. Build lldb/lldb-dap as above (patched, Python off).
2. `lldb-dap` DAP session: `initialize`, then `launch` with
`program = UnrealEditor-Cmd.exe`, `args = [, -nullrhi, -unattended, -nosound, -NOSPLASH]`,
`initCommands = ["settings set symbols.load-on-demand true", "log enable -T -f lldb process state thread event", "log enable -T -f windows event exception process thread break"]`.
3. `setBreakpoints` at a source line in the game module, `configurationDone`.
4. Wait. The `stopped(breakpoint)` event never arrives; the adapter is alive but the target does not
progress.

## Observed behavior (instrumented)
Measured with (a) a non-invasive `cdb -pv` snapshot of the lldb-dap process at t≈58/100/160 s,
(b) `.time`/`!runaway` CPU accounting across those snapshots, (c) the lldb `windows` log channel
(exception/event), and (d) editor thread `WaitReason` sampling. n=2 runs, cross-consistent.

- **lldb debug loop is healthy and idle.** At the steady wedge (t=102 s and t=161 s, two consecutive
samples) the DebuggerThread is parked in `ntdll!NtWaitForDebugEvent` ← `KERNELBASE!WaitForDebugEventWorker`.
Its CPU time is **byte-identical** across the 59 s between those samples (Kernel/User/per-thread
times unchanged) → the loop processed zero events in that window and is fully idle. It had finished
its symbol work by ~t=92 s (RSS plateau).
- **All first-chance exceptions were handled normally.** 63 exception round-trips, all resolved
(`ExceptionPred`): 62 masked/passed-to-app (`= 2`), 1 loader breakpoint (`= 1`). No stuck/unresolved
exception; late thread create/exit events are still `ContinueDebugEvent`-ed after the wedge.
- **The target self-Waits.** Editor threads show `WaitReason` = UserRequest / ExecutionDelay /
EventPairLow / Executive (their own blocking waits). **Zero threads are `Suspended`** during the
wedge window. As a positive control, *pre-wedge* (t≈6–93 s) the sampler does observe `Suspended`
rising 3 → 66 (the target is debug-stopped during module loading) — so the instrument works and the
in-wedge "0 Suspended" is a real self-Wait, not a measurement gap.

Conclusion: at wedge time lldb has continued the target and is correctly waiting for the next debug
event that never comes; the target has deadlocked inside its own initialization.

## Contrast (honest, not a controlled comparison)
The portable Windows debugger `cdb` (10.0.28000) launches the *same* target and reaches the game tick
in **15.1 s**. However this used `sxi *` (ignore first-chance exceptions), lazy symbol loading, and no
breakpoint — **three uncontrolled variables** vs the lldb run. So this establishes only that the
wedge is *specific to this lldb configuration*, not a clean cdb-vs-lldb result. A cdb run with default
exception disposition was not performed.

Additionally, Visual Studio 2022 (devenv, Concord engine) attached to the same target **~0.7 s after
process spawn** — with the same source breakpoint set — observed the rest of init and hit the first
game tick (tick counter = 1) at ~17 s from spawn, **no wedge (n=2)**. Caveats: this is early-attach,
not a true launch (the earliest loader window ran undebugged), and the exception disposition was
inherited from that machine's VS profile rather than explicitly controlled — so it aligns only the
breakpoint variable. It adds a second same-machine debugger that observes most of UE init without
wedging; it is still not a controlled lldb-vs-X comparison. (Assets: `vsdbg/vs2_devenv_probe.ps1`,
`vsdbg/vs2_earlyattach_run{1,2}_*` in the repro repo below.)

## What is NOT established (disclosures)
- Observed only on the **patched + LOD-on** build. No unpatched + LOD-on control run exists on either
bed (unpatched lldb balloons out of memory before reaching this init stage on 32 GB). So we cannot
say the patch is counterfactually innocent — only that at wedge time the debug loop is idle/healthy
(patch's lazy hydration is *not* actively wedging the event thread).
- The pre-wedge symbol-materialization I/O (the DebuggerThread was seen in `CreateFileW`/PDB load
during t≈50–90 s) is a candidate contributor to the init slowdown, and hydration completion (~t92 s)
overlaps the wedge onset in time — this instrumentation did not exclude that overlap as a factor.
- liblldb was a Release build (no PDB), so cdb resolves OS wait frames but not internal liblldb frame
names.

## Candidate mechanism (unconfirmed)
Serialized handling of ~862 module-load events plus 63 first-chance exceptions through the single
Windows DebuggerThread stretches init from ~40 s (no debugger) to ~92 s; a UE init timeout / thread
join / watchdog may then trip and self-deadlock. Exact trigger not isolated.

## Workaround
**Attach after the editor has finished initializing** — the attach path reaches the same debugging
state (breakpoint hit, `frame variable`, native rendering) without the wedge (measured attach ≈ 82 s
on the same bed). **End-to-end, a cold editor spawn → boot-complete detection (game-module tick) →
attach → first game-code breakpoint hit measured 105–107 s (n=2) on this bed** — well inside a 300 s
usability budget (caveat: AdePoc-minimal project, process-cold but DDC/OS-cache warm; a real game's
slower boot + heavier symbols would eat into that headroom). Recommended until the wedge is understood
(note: post-attach surfaces — dynamic module load, exception storms, GUI interaction — were not
soak-tested; an intermittent disconnect AV is reported separately, `upstream/draft-issue-detach-av.md` in the repro repo).

## Attach-path recurrence (intermittent 1/4, mechanism unresolved)
The attach workaround is **not fully immune** to the same *phenotype*. Driving a heavy asset-editor
open (a Niagara system, created + opened from the editor's Python) **while attached**, the game tick
froze permanently once — same signature as the launch wedge (game thread stops ticking, editor process
stays alive, survives detach). But this is **intermittent (1 of 4 attach + Niagara-open runs)** and did
**not** reproduce in the 3 subsequent runs — one with base log channels, **two with the full
instrumentation suite** above (symbol + `windows` exception/event channels) — all three of which ran
the full driver (Niagara open → Blueprint open → full PIE via console `PlayInEditor` → PIE Simulate)
with continuous ticking and no freeze. Because the freeze never recurred *while instrumented*, **no cdb
stack / WaitReason / windows-channel snapshot of the attach-path freeze was captured** — so whether it
shares this wedge's mechanism (target self-Wait, debug loop idle) is **undetermined**.

**No determinism claim.** A no-lldb control (the same Niagara-open driver, no debugger) showed only a
**4.4 s transient hitch** and completed all scenarios — so the freeze requires lldb's presence but is
**not a deterministic consequence of it** (1/4 under attach). The instrumentation-channel difference was
excluded as the cause (the two no-freeze instrumented runs used the exact original symbol channel);
run-to-run nondeterminism (shader / thread / first-chance timing) is the likely factor. Surface-level
observation across the attach runs (obs window ≤ 60 s): Niagara-open 3/3 clean, Blueprint 3/3, full PIE
3/3, PIE Simulate 2/3, full-schedule completion 0/3 (window shorter than the driver schedule). Assets:
`realbed/realbed_h17ctl_*` (no-lldb control), `realbed/realbed_h17b_run{1,2,3}_*` (attach repro + instrumentation).

Reproduction assets: `realbed/` in the companion repro repo
https://github.com/cherleey/lldb-ue-pdb-repro (probe harnesses `realbed_h9_run.ps1`,
per-run `realbed_h9_run{1,2}_windows.txt`, `_cdb_s*`, `_waitreason.csv`, `_result.json`).

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction assets in realbed/ and the lldb-ue-pdb-repro harnesses, then reproduce the launch using the stated symbols.load-on-demand and breakpoint settings. Read the Windows event logs, WaitReason samples, and lldb process/thread logs around SessionServices.dll loading, focusing on the DebuggerThread and target self-Waits. Done means the trigger is isolated and the launch no longer wedges under the reproduced configuration.

Written by the indexing model from the issue text.

Assessment

Domain
devtools, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.