[clang] MSVC mangler hits UNREACHABLE "Invalid type expected" (mangleAutoReturnType) on lambda conversion to function pointer with deduced return type
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
When targeting the MSVC ABI, clang crashes while mangling a lambda's conversion operator to function pointer when the lambda has a deduced return type (`-> auto &&`): the conversion operator's return type is a pointer to a *function* type carrying the placeholder, and `MicrosoftCXXNameMangler::mangleAutoReturnType`'s dispatcher only handles `MemberPointer`/`Pointer`/`LValueReference`/`RValueReference`, so it falls into `llvm_unreachable("Invalid type expected")`.
In builds with assertions this is a clean UNREACHABLE abort. In release builds without assertions the unreachable is UB, and we observed it as a hard segfault (0xC0000005 on Windows) or an infinite loop / time-limit kill, depending on the build — which makes it easy to misdiagnose as an OOM or CI timeout.
## Reproducer
https://gcc.godbolt.org/z/vs3MEYP61 (clang assertions trunk, `-std=c++20 --target=x86_64-pc-windows-msvc`)
```cpp
template void add(F);
template
struct RegisterOneType
{
inline static const decltype(nullptr) register_type = (add(+[](int &x) -> auto && { return x; }), nullptr);
};
inline void bake() { (void)&RegisterOneType::register_type; }
```
```
clang++ -std=c++20 --target=x86_64-pc-windows-msvc -c repro.cpp
```
Note `bake()` is never called; referencing `®ister_type` from it is enough to trigger the end-of-TU instantiation and codegen of the inline variable's dynamic initializer.
## Output (assertions trunk)
```
Invalid type expected
UNREACHABLE executed at /root/llvm-project/llvm/tools/clang/lib/AST/MicrosoftMangle.cpp:2572!
...
2. :6:43: instantiating variable definition 'RegisterOneType::register_type'
3. :6:43: Generating code for declaration 'RegisterOneType::register_type'
4. :6:65: Mangling declaration 'RegisterOneType::(lambda)::operator int &(*)(int &)'
#9 0x000000000835889a (anonymous namespace)::MicrosoftCXXNameMangler::mangleAutoReturnType(clang::QualType, (anonymous namespace)::MicrosoftCXXNameMangler::QualifierMangleMode) MicrosoftMangle.cpp:0:0
#10 0x000000000835866b (anonymous namespace)::MicrosoftCXXNameMangler::mangleAutoReturnType(clang::QualType, (anonymous namespace)::MicrosoftCXXNameMangler::QualifierMangleMode) MicrosoftMangle.cpp:0:0
#11 0x0000000008358db7 (anonymous namespace)::MicrosoftCXXNameMangler::mangleFunctionType(clang::FunctionType const*, clang::FunctionDecl const*, bool, bool) MicrosoftMangle.cpp:0:0
#12 0x0000000008359fe9 (anonymous namespace)::MicrosoftCXXNameMangler::mangleType(clang::QualType, clang::SourceRange, (anonymous namespace)::MicrosoftCXXNameMangler::QualifierMangleMode) MicrosoftMangle.cpp:0:0
#13 0x000000000835a268 (anonymous namespace)::MicrosoftCXXNameMangler::mangleType(clang::QualType, clang::SourceRange, (anonymous namespace)::MicrosoftCXXNameMangler::QualifierMangleMode) MicrosoftMangle.cpp:0:0
#14 0x0000000008359214 (anonymous namespace)::MicrosoftCXXNameMangler::mangleFunctionType(clang::FunctionType const*, clang::FunctionDecl const*, bool, bool) MicrosoftMangle.cpp:0:0
#15 0x0000000008361235 (anonymous namespace)::MicrosoftMangleContextImpl::mangleCXXName(clang::GlobalDecl, llvm::raw_ostream&) MicrosoftMangle.cpp:0:0
#16 0x0000000008342d9c clang::MangleContext::mangleName(clang::GlobalDecl, llvm::raw_ostream&)
#17 0x0000000004aac3b5 getMangledNameImpl(clang::CodeGen::CodeGenModule&, clang::GlobalDecl, clang::NamedDecl const*, bool) CodeGenModule.cpp:0:0
```
## Versions
Verified on Compiler Explorer with `-std=c++20 --target=x86_64-pc-windows-msvc`:
| version | result |
|---|---|
| clang 18.1.0 | compiles OK |
| clang 19.1.0 | compiles OK |
| clang 20.1.0 (assertions) | UNREACHABLE at MicrosoftMangle.cpp:2551 |
| clang 20.1.0 / 21.1.0 (release) | hangs until killed |
| clang 22.1.0 (assertions) | UNREACHABLE at MicrosoftMangle.cpp:2572 |
| clang 22.1.x (release, Windows native) | segfault (0xC0000005) |
| trunk (assertions) | UNREACHABLE at MicrosoftMangle.cpp:2572 |
So this looks like a regression introduced with the placeholder-return-type mangling for MSVC 1920+ (#102848, relanded as #104722, first shipped in clang 20).
The same TU compiles fine with the Itanium mangler (e.g. `--target=x86_64-pc-linux-gnu`).
`-std=c++23` is affected the same way.
## Context
Originally hit in MeshLib's Python-binding CI (pybind11-style registration code), where it appeared only after `-fpch-instantiate-templates` started instantiating these lambdas inside a PCH — every fragment compile then died with 0xC0000005 inside `clang::MicrosoftMangleContext`. The reduced reproducer above needs no PCH.
Contributor guide
Research direction
Start in clang/lib/AST/MicrosoftMangle.cpp at MicrosoftCXXNameMangler::mangleAutoReturnType and reproduce with the provided C++20 command targeting x86_64-pc-windows-msvc. Trace the lambda conversion operator's function-pointer return type through the mangler, then add regression coverage for the reproducer and verify that assertion and release builds compile without crashing, hanging, or reaching the unreachable path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100