llvm / llvm/llvm-project

[X86][WinEH] Win64 catchret continuation resumes with stale RSP after dynamic alloca + throwing invoke

Open
#191,615 1 comment 0 reactions 0 assignees View on GitHub
backend:X86 platform:windows
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

## Description

Consider the following reduced IR:

```llvm
declare i32 @__CxxFrameHandler3(...)
declare void @opaque()
declare void @maythrow()
declare void @maythrow_big(i64, i64, i64, i64, i64, i64, i64, i64, i64)
declare ptr @llvm.stacksave.p0()
declare void @llvm.stackrestore.p0(ptr)
declare void @llvm.memset.p0.i64(ptr nocapture writeonly, i8, i64, i1)

define void @test(i64 %n) personality ptr @__CxxFrameHandler3 {
entry:
%sp = call ptr @llvm.stacksave.p0()
%buf = alloca i8, i64 %n, align 16
call void @llvm.memset.p0.i64(ptr %buf, i8 0, i64 %n, i1 false)
call void @llvm.stackrestore.p0(ptr %sp)
invoke void @maythrow() to label %exit unwind label %catch.dispatch

catch.dispatch:
%cs = catchswitch within none [label %catch] unwind to caller

catch:
%cp = catchpad within %cs [ptr null, i32 64, ptr null]
call void @opaque() [ "funclet"(token %cp) ]
catchret from %cp to label %cont

cont:
call void @opaque()
ret void

exit:
ret void
}

define void @test_manyargs(i64 %n) personality ptr @__CxxFrameHandler3 {
entry:
%sp = call ptr @llvm.stacksave.p0()
%buf = alloca i8, i64 %n, align 16
call void @llvm.memset.p0.i64(ptr %buf, i8 0, i64 %n, i1 false)
call void @llvm.stackrestore.p0(ptr %sp)
invoke void @maythrow_big(i64 0, i64 1, i64 2, i64 3, i64 4, i64 5, i64 6,
i64 7, i64 8)
to label %exit unwind label %catch.dispatch

catch.dispatch:
%cs = catchswitch within none [label %catch] unwind to caller

catch:
%cp = catchpad within %cs [ptr null, i32 64, ptr null]
call void @opaque() [ "funclet"(token %cp) ]
catchret from %cp to label %cont

cont:
call void @opaque()
ret void

exit:
ret void
}
```

On `x86_64-pc-windows-msvc`, the parent frame uses explicit `ADJCALLSTACKDOWN/UP` around the throwing `invoke` because the function has a dynamic alloca and therefore does not reserve a fixed outgoing call frame.

Before the matching `ADJCALLSTACKUP` runs, the exceptional edge transfers control into the funclet. `catchret` then resumes the continuation block without first restoring RSP to the parent frame's stable post-prologue value.

This means the continuation inherits the throwing invoke's transient call-frame adjustment.

In practice, a real C++ reproducer using `_alloca()` inside a loop accumulates stack drift on every throw/catch iteration:

```text
diff(0-100) = 3200
```

and if the throwing call needs an 80-byte outgoing call frame instead of 32 bytes, the drift scales accordingly:

```text
diff(0-100) = 8000
```

## Expected behavior

`catchret` should resume through a block that restores RSP to the parent frame's stable post-prologue value before executing the continuation block, independent of the outgoing call-frame size used by the throwing invoke.

## Actual behavior

The generated Win64 code resumes the continuation with the invoke's temporary call-frame adjustment still in effect, so the continuation runs with a stale RSP and repeated throws drift the stack.

## Assembly shape

For the 32-byte case, the generated parent frame looks like:

```asm
sub rsp, 32
call maythrow
add rsp, 32
```

but on the exceptional edge the `add rsp, 32` is not reached. The catch funclet returns directly to the continuation, so the continuation executes without normalizing RSP first.

The 80-byte case behaves the same way with `sub/add rsp, 80`.

## Root cause

This looks like a gap in how Win64 `catchret` continuations are modeled:

* the parent frame does not reserve a fixed call frame when dynamic allocas are present,
* the throwing `invoke` therefore uses explicit call-frame adjustments,
* the exceptional edge leaves before the matching destroy runs,
* but Win64 `catchret` continuations do not restore the parent stack state before resuming normal control flow.

The bug is not in funclet fixed-frame sizing. The leak size tracks the throwing invoke's outgoing call-frame size instead.

## Versions

I reproduced this with:

* LLVM/clang 19
* LLVM/clang 21
* LLVM/clang thunk

## Proposed fix

I have a local fix that:

* always routes Win64 `catchret` through a dedicated restore block, and
* restores RSP from the parent frame pointer's stable Win64 SEH frame offset before entering the continuation.

I also added a regression test covering both 32-byte and 80-byte outgoing call-frame sizes.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reduced LLVM IR on x86_64-pc-windows-msvc and inspect the generated assembly around the throwing invoke, catchret, and explicit ADJCALLSTACK adjustments. Compare the 32-byte and 80-byte cases. Done means catchret reaches the continuation with the parent frame's stable RSP and the regression test covers both outgoing call-frame sizes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.