llvm / llvm/llvm-project

`ext_vector_type` of fp80 `long double` miscompiled with stack smashing protection (SSP) or ASAN

Open
#212,002 3 comments 0 reactions 0 assignees View on GitHub
miscompilation
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Given following program

```cpp
#include

typedef long double LongDouble3Vec __attribute__((ext_vector_type(3)));
static_assert(sizeof(LongDouble3Vec) == 64);

int main(int, const char**) {
LongDouble3Vec v1;
memset(&v1, 0xAB, sizeof(LongDouble3Vec));
}
```

ASAN gives

```
This frame has 1 object(s):
[64, 96) 'v1' (line 8) <== Memory access at offset 96 overflows this variable
```

The size of `LongDouble3Vec` is 64 bytes. But ASAN seems to believe it is 32 bytes. Note that without ASAN, we will see

```
alloca <3 x x86_fp80>
```

The layout of this vector is first 3 x 80 bits contains the data and from bit 240 to bit 511 are padding bits. perhaps ASAN believes 3 x 80 bits == 240 bits and round up to power of 2 is 256 bits (32 bytes?)

https://godbolt.org/z/nssvWG91G

Contributor guide

Open the contributing guide

Research direction

Start by compiling the supplied C++ reproducer with ASAN and stack smashing protection, then compare the sanitizer's reported object size with sizeof(LongDouble3Vec) and the non-ASAN alloca. Trace how the compiler models ext_vector_type vectors of x86_fp80 during stack-object instrumentation. Done means the 64-byte vector is instrumented consistently and the memset no longer reports an overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.