llvm / llvm/llvm-project

clang -O3 codegen mishandles a stack-top/guard pattern using __builtin_alloca(0) plus a later dynamic __builtin_alloca(bytes)

Open
#202,846 1 comment 0 reactions 0 assignees View on GitHub
llvm:optimizations
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Look at the below reproducer.

```c
#include

extern size_t guard_size(void);
extern int reuse_guard(void *old_top);
extern void install_guard(void *guard_start, size_t bytes, int reuse);
extern void use_live(void *live);

#define STACK_ALLOC(bytes) __builtin_alloca(bytes)

__attribute__((noinline))
int repro_tiny_no_touch(int seed)
{
char live[48];
size_t bytes = guard_size();
void *old_top = STACK_ALLOC(0);
int reuse;

live[0] = (char)seed;
live[47] = (char)(seed + 1);

reuse = reuse_guard(old_top);
if (!reuse)
{
void *tmp = STACK_ALLOC(bytes);
if (tmp)
old_top = (void *)((char *)old_top - bytes);
}

install_guard(old_top, bytes, reuse);
use_live(live);

return live[0] + live[47] + reuse;
}

__attribute__((noinline))
int repro_tiny_with_touch(int seed)
{
char live[48];
size_t bytes = guard_size();
void *old_top = STACK_ALLOC(0);
int reuse;

live[0] = (char)seed;
live[47] = (char)(seed + 1);

reuse = reuse_guard(old_top);
if (!reuse)
{
void *tmp = STACK_ALLOC(bytes);
if (tmp)
{
*(volatile char *)tmp = 0;
old_top = (void *)((char *)old_top - bytes);
}
}

install_guard(old_top, bytes, reuse);
use_live(live);

return live[0] + live[47] + reuse;
}
```

There are 2 experiments, 'with_touch' and 'no_touch' functions.

1. no_touch experiment: this case is useful but less decisive. Pairwise IR shows InstCombinePass removes the dynamic alloca.
Before InstCombine, %1 = alloca i8, i64 %call exists. After InstCombine, it is gone and only
%0 = alloca [0 x i8] remains. This may be debated because the allocated pointer is not touched.

2. with_touch experiment: this removes that ambiguity. Pairwise IR shows InstCombine keeps the dynamic alloca because of the volatile
store: %2 = alloca i8, i64 %call and store volatile i8 0, ptr %2 remain after InstCombine.
The first concrete bad frame layout appears during Prologue/Epilogue Insertion & Frame Finalization: before the pass,
frame objects are symbolic (fi#0, fi#1, fi#2) ; after the pass, fi#0 is [SP-40] and fi#1 is [SP-88] ,
producing old_top = rbp - 32 and live = rbp - 80..rbp - 33.

The no_touch shows a related optimization concern, but with_touch demonstrates a stronger codegen bug:
real dynamic stack movement exists, yet the captured old stack top is lowered to a fixed frame address that lets the guard overwrite a still-live local.

Required behavior:

The compiler must keep the guard range inside the dynamically allocated stack area, not inside any still-live fixed-frame local.
For with_touch, the required lowering is:

1. Place live[48] in the fixed frame.
2. Capture old_top so that it represents the real stack top below fixed locals.
3. Perform dynamic stack allocation:
rsp = rsp - align(bytes)
4. Compute guard_start from the safe old_top / dynamic allocation boundary:
guard_start = old_top - bytes
5. Call:
install_guard(guard_start, bytes, reuse)
6. Ensure:
[guard_start, guard_start + bytes) does not overlap live[48]

For the current bad -O3 output:

live[48] = [rbp - 80, rbp - 32)
old_top = rbp - 32
guard_start = rbp - 32 - bytes

For bytes = 48, that overlaps live[48]. Required behavior is that old_top must be below live[48], for example:

live[48] = [rbp - 80, rbp - 32)
old_top <= rbp - 80
guard_start = old_top - bytes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.