`ext_vector_type` of fp80 `long double` miscompiled with stack smashing protection (SSP) or ASAN
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Given following program
```cpp
#include
typedef long double LongDouble3Vec __attribute__((ext_vector_type(3)));
static_assert(sizeof(LongDouble3Vec) == 64);
int main(int, const char**) {
LongDouble3Vec v1;
memset(&v1, 0xAB, sizeof(LongDouble3Vec));
}
```
ASAN gives
```
This frame has 1 object(s):
[64, 96) 'v1' (line 8) <== Memory access at offset 96 overflows this variable
```
The size of `LongDouble3Vec` is 64 bytes. But ASAN seems to believe it is 32 bytes. Note that without ASAN, we will see
```
alloca <3 x x86_fp80>
```
The layout of this vector is first 3 x 80 bits contains the data and from bit 240 to bit 511 are padding bits. perhaps ASAN believes 3 x 80 bits == 240 bits and round up to power of 2 is 256 bits (32 bytes?)
https://godbolt.org/z/nssvWG91G
Contributor guide
Research direction
Start by compiling the supplied C++ reproducer with ASAN and stack smashing protection, then compare the sanitizer's reported object size with sizeof(LongDouble3Vec) and the non-ASAN alloca. Trace how the compiler models ext_vector_type vectors of x86_fp80 during stack-object instrumentation. Done means the 64-byte vector is instrumented consistently and the memset no longer reports an overflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100