llvm / llvm/llvm-project

MachineOutliner corrupts WinEH (SEH) tables for MSVC C++ EH funclets

Open
#210,009 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

miscompilation platform:windows
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Title

MachineOutliner corrupts WinEH (SEH) tables for MSVC C++ EH funclets, producing invalid .seh_startepilogue/.seh_endepilogue sequences

Environment

  • clang version 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211)
  • Target: x86_64-pc-windows-msvc
  • Reproduces via both clang-cl and plain clang++ --target=x86_64-pc-windows-msvc

Summary

With the machine outliner enabled (-mllvm --enable-machine-outliner) and no LTO, compiling
a small C++ file with two structurally-identical functions that each contain multiple local
objects with non-trivial (potentially-throwing) constructors fails with the integrated assembler
rejecting the emitted WinEH directives:

error: starting epilogue (.seh_startepilogue) before prologue has ended (.seh_endprologue) in ?make0@@YA?AUPair@@PEBD0@Z
error: Stray .seh_endepilogue in ?make0@@YA?AUPair@@PEBD0@Z
error: starting epilogue (.seh_startepilogue) before prologue has ended (.seh_endprologue) in ?dtor$30@?0??make0@@YA?AUPair@@PEBD0@Z@4HA
error: Stray .seh_endepilogue in ?dtor$30@?0??make0@@YA?AUPair@@PEBD0@Z@4HA
...
error: .seh_ directive must appear within an active frame
16 errors generated.

The errors hit both the two visible functions and their compiler-generated partial-construction
cleanup funclets (dtor$N — the per-scope unwind handlers MSVC C++ EH emits when an earlier local's
constructor must be undone if a later local's constructor throws). This strongly suggests the
outliner is extracting a code sequence that spans (or otherwise interacts badly with) a WinEH
funclet boundary, without correctly updating/duplicating the per-function SEH state-transition
metadata for the extracted/rewritten code.

  • --hot-cold-split alone (without the outliner) does not reproduce this — isolated to
    --enable-machine-outliner specifically.
  • Reproduces at -O1, -O2, and -Os; does not reproduce at -O0 (outlining presumably
    doesn't kick in without any optimization).
  • Requires no LTO and no project-specific flags beyond /EHsc (or -fexceptions -fcxx-exceptions) + the outliner flag.
  • We found this via a real-world build (an internal C++ codebase) where a build-config change
    removed LTO from a couple of targets that happened to still have --enable-machine-outliner
    enabled; under our normal (non-LTO-backend-forwarded) LTO configs the outliner pass never ran
    against final WinEH-table-bearing code (our -mllvm enable is only passed at the front-end
    compile step, never forwarded to the LTO backend link step), which is presumably why this had
    gone unnoticed — losing LTO on those targets promptly hit this.
  • This is not LTO-mode-specific. Forcing the same flag into the LTO backend itself (rather
    than just the front-end compile step) reproduces the identical corruption under both ThinLTO
    and full LTO:
    clang++ --target=x86_64-pc-windows-msvc -fms-extensions -fexceptions -fcxx-exceptions \
        -std=c++20 -O2 -flto=thin -fuse-ld=lld -Wl,-mllvm:--enable-machine-outliner \
        minimal_main.cpp -o out.exe
    
    This hits the exact same .seh_startepilogue/Stray .seh_endepilogue errors, this time inside
    libcpmt.lib's own EH vtable thunks (std::exception's and std::bad_alloc's scalar/vector
    deleting dtors) once they get pulled into the same LTO merge as our TU. Same result with
    -flto=full. So the bug is in the pass/codegen interaction itself, not something no-LTO-specific
    — it just happens that most projects' LTO configs (like ours, until now) never forward
    -mllvm outliner/hot-cold-split enables to the LTO backend, so it goes unnoticed there.

Minimal reproducer (17 lines, self-contained, only depends on <string>)

#include <string>

struct Str {
    Str( const char * s ) : s_( s ) {}
    Str( const Str & o ) : s_( o.s_ ) {}
    std::string s_;
};

struct Pair {
    Pair( Str const & a, Str const & b ) : a_( a ), b_( b ) {}
    Str a_;
    Str b_;
};

Pair make0( char const * x, char const * y ) { Str a( x ); Str b( y ); return Pair( a, b ); }
Pair make1( char const * x, char const * y ) { Str a( x ); Str b( y ); return Pair( a, b ); }

Command line:

clang++ -c --target=x86_64-pc-windows-msvc -fms-extensions -fexceptions -fcxx-exceptions \
    -std=c++20 -O2 -mllvm --enable-machine-outliner minimal.cpp -o minimal.o

(equivalently: clang-cl /c /EHsc /std:c++20 -O2 -mllvm --enable-machine-outliner minimal.cpp)

A single copy of make0 alone (no make1) does not reproduce — the outliner needs at least
two occurrences of the sequence to consider extracting it, consistent with this being a genuine
outlining-vs-WinEH interaction rather than something specific to one function's shape.

Expected behavior

The file should compile cleanly (as it does at -O0, or with the outliner disabled at any
optimization level).

Notes

  • 100% deterministic across repeated runs on the same machine.
  • Not investigated: whether this is specific to the MSVC C++ EH model (/EHsc, WinEH funclets)
    vs. also reproducible with Itanium EH tables on a non-Windows triple — have not tried a
    Linux/macOS target with the outliner enabled + LTO disabled + this same code shape.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by compiling the provided minimal.cpp with the x86_64-pc-windows-msvc command and --enable-machine-outliner to reproduce the WinEH directive errors. Then inspect the MachineOutliner and WinEH interaction around the affected visible functions and dtor$N funclets. Done means the reproducer compiles cleanly at optimization levels with the outliner enabled, without invalid .seh sequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.