llvm / llvm/llvm-project

[X86] Mixing Win64 and SYSV calling conventions with -mno-sse can lead to silent miscompilation

Open
#196,908 3 comments 0 reactions 0 assignees View on GitHub
ABI backend:X86 miscompilation
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

# Description

Quoting from the [Microsoft documentation for the X64 ABI](https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#callercallee-saved-registers):
> The (*Windows*) x64 ABI considers registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, R15, **and XMM6-XMM15** _nonvolatile_. **They must be saved and restored by a function that uses them.**

However I found a case where this is not (and maybe cannot be, details later) honored by LLVM.

For the SYSV 64 (e.g. Linux) ABI the caller must save these register, not the callee. Mixing of these two calling conventions is allowed, the backend allows declaring or defining functions that use the Windows 64 calling convention when compiling for a Linux triple, and the other way around too.
When a function is defined to be using the Windows ABI, in order to conform to the ABI, it **must** save XMM6-XMM15 when making a call to a SYSV64 function. This is because the called function _might_ modify the registers, and according to **its** ABI it does NOT have to save them if it does.

LLVM usually implements this correctly, however to save XMM registers SSE instructions have to be used. If the function body is compiled with `-mno-sse` then it is not possible to do this, and the backend silently omits saving these registers.

# Relevant Issues

https://reviews.llvm.org/D82562 solved a similar case of the ABI changing for AVX / NO AVX, but it was narrower in scope because it only affected function parameters and returns, not callee saved registers.

# Impact

When the compilation targets Windows, but there is a call to SYSV64 ABI function, the impact is limited to that call-site. The caller function might fail to uphold its ABI contract, and its callers might see SSE registers being corrupted.

In the other case when the default calling convention is Linux, but a function is explicitly defined as using the Windows ABI, then impact is more severe. The compiler might implicitly emit calls to runtime functions (e.g. memset, memcpy, compiler-rt builtins etc) and these "hidden" calls can also corrupt XMM registers silently.

While researching this issue, I tried to find where such mixing of ABIs is used:
GCC and Clang support [`__attribute__((ms_abi))`](https://clang.llvm.org/docs/AttributeReference.html#ms-abi) and [`__attribute__((sysv_abi))`](https://clang.llvm.org/docs/AttributeReference.html#sysv-abi)
The main use-cases I was able to find where RPC / foreign function interfaces and translation layers (like WINE), and code interacting with UEFI routines / callbacks.

For UEFI use-cases it is highly likely that both sides of the call have SSE disabled, therefore this issue cannot be observed. FFI layers and translations often have hand-written assembly code to save not just CSR registers but more.

# RFC

Based on the fact that real world use is limited, and the attributes could be argued to be for expert users already I believe a documentation only fix might be reasonable. It's not clear what else LLVM can do here, except maybe warn when it detects such an issue. However for the current users of these attributes they would be basically be all false positives.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.