[PowerPC64] GETtlsADDR/GETtlsldADDR omit FP/VSX/VMX clobbers, miscompiling dynamic-TLS stores on FreeBSD
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
## Summary
On `powerpc64le-unknown-freebsd`, clang miscompiles any shared object that stores
a vector/FP value into a `__thread` variable. The generated code materialises the
value into a **volatile** VSX register *before* `bl __tls_get_addr` and consumes it
*after* the call, so the value returned by the dynamic loader's internal work is
stored into the TLS variable instead.
The cause is that the GD/LD TLS pseudos declare only GPR/LR/CTR/CR clobbers:
`llvm/lib/Target/PowerPC/PPCInstr64Bit.td` (still current on `main`):
```
let hasExtraSrcRegAllocReq = 1, hasExtraDefRegAllocReq = 1 in {
// LR8 is a true define, while the rest of the Defs are clobbers. X3 is
// explicitly defined when this op is created, so not mentioned here.
// This is lowered to BL8_NOP_TLS by the assembly printer, so the size must be
// correct because the branch select pass is relying on it.
let Defs = [X0,X4,X5,X6,X7,X8,X9,X10,X11,X12,LR8,CTR8,CR0,CR1,CR5,CR6,CR7], Size = 8 in
def GETtlsADDR : GETtlsADDRPseudo <"#GETtlsADDR">;
...
let Defs = [X0,X4,X5,X6,X7,X8,X9,X10,X11,X12,LR8,CTR8,CR0,CR1,CR5,CR6,CR7], Size = 8 in
def GETtlsldADDR : GETtlsldADDRPseudo <"#GETtlsldADDR">;
```
No FPRs, no VSRs, no VRs. The register allocator is therefore free to keep a value
in e.g. VSR0 (= FPR0, volatile) across the call.
That assumption is only sound when the linker interposes a register-saving stub —
GNU ld's `--tls-get-addr-regsave`, which exists precisely because `__tls_get_addr`
does clobber volatile registers, and which GNU ld enables when glibc advertises
`__tls_get_addr_opt`. On FreeBSD none of that machinery is present:
* `rtld-elf` has no `__tls_get_addr_opt` (`nm -D /libexec/ld-elf.so.1 | grep -c
tls_get_addr_opt` → `0`), and the produced object has `DT_PPC64_OPT = 0x0`.
* lld emits a plain PLT stub that saves only `r2`:
```
00000000000108a0 <__plt___tls_get_addr>:
108a0: std 2, 24(1)
108a4: addis 12, 2, 1
108a8: ld 12, -32688(12)
108ac: mtctr 12
108b0: bctr
```
* `libexec/rtld-elf/powerpc64/reloc.c` implements `__tls_get_addr` as ordinary C
(`return tls_get_addr_common(_tcb_get(), ti->ti_module, ti->ti_offset + TLS_DTV_OFFSET);`),
so it clobbers volatiles like any other C function.
GCC does not make this assumption — it materialises constants into GPRs *after* the
call — so this is clang-specific.
## Reproducer
```c
/* lib.c ; cc -O2 -fPIC -shared -o libtlsbug.so lib.c */
#include
struct big { void *a; void *b; uint64_t c; uint64_t d; };
__thread struct big g;
void init_it(void) { g.a = 0; g.b = 0; g.c = 0x400000; g.d = 0; }
void *get_a(void) { return g.a; }
void *get_b(void) { return g.b; }
```
```c
/* main.c ; cc -O2 -o t main.c -L. -ltlsbug -Wl,-rpath,$PWD */
#include
void init_it(void); void *get_a(void); void *get_b(void);
int main(void) {
init_it();
printf("a=%p b=%p %s\n", get_a(), get_b(),
(get_a() == 0 && get_b() == 0) ? "OK" : "*** CLOBBERED ***");
return get_a() != 0 || get_b() != 0;
}
```
Expected `a=0x0 b=0x0 OK`; actual:
```
a=0x8100862c0 b=0x810030300 *** CLOBBERED ***
```
`0x8100862c0` is `&obj_rtld`, an internal rtld object — i.e. leftover contents of
VSR0 after the loader ran.
### Generated code
```
0000000000106f0 :
106f0: addis 2, 12, 2
106f4: addi 2, 2, -31744
106f8: mflr 0
106fc: stdu 1, -32(1)
10700: std 0, 48(1)
10704: addis 3, 2, 0
10708: xxlxor 0, 0, 0 <-- VSR0 = 0, materialised BEFORE the call
1070c: addi 3, 3, -32760
10710: bl 0x108a0 <__plt___tls_get_addr>
10714: ld 2, 24(1)
10718: lis 4, 64
1071c: stxvd2x 0, 0, 3 <-- stores clobbered VSR0 into g.a/g.b
10720: std 4, 16(3)
...
```
Confirmed under gdb: `$f0 == 0x0` immediately before the `bl`, and
`$f0 == 0x8100862c0` immediately after it.
## Notes on when it bites
Only the **first** `__tls_get_addr` call for a given module is corrupted:
```
call 1: a=0x8100862c0 b=0x810030300 *** CLOBBERED ***
call 2: a=0x0 b=0x0 OK
```
FreeBSD's `tls_get_addr_common` has a GPR-only fast path; the first call takes the
slow path (`allocate_module_tls`, `memcpy`) which uses VSX and destroys VSR0. This
makes the bug intermittent-looking and very hard to attribute — it surfaced here as
a `free()` of an rtld-internal pointer during teardown in `lang/janet`, whose
`JanetVM` is a `__thread` struct whose first field pair is zeroed exactly this way.
## Affected configurations
| variant | result |
| --- | --- |
| `-O0` | OK (no hoisting across the call) |
| `-O1`, `-O2` | **clobbered** |
| `-O2 -mcpu=power9` | **clobbered** |
| `-O2 -mno-vsx` | OK |
| static executable (local-exec TLS, no call) | OK |
| gcc 14 `-O2` | OK |
* clang 21.1.8 (`llvmorg-21.1.8-0-g2078da43e25a`) and clang 19.1.7 both affected,
so this is not a recent regression.
* Target `powerpc64le-unknown-freebsd16.0`, hardware POWER9. FreeBSD 15.1 emits
byte-identical (miscompiled) code but happens not to trip the loader's slow path
at the first access, so it does not manifest there.
## Suggested fix
Add the volatile FP/VSX/VMX registers to the `Defs` lists of `GETtlsADDR`,
`GETtlsldADDR` and the PCREL variants, at least when the target does not have the
`__tls_get_addr_opt` / register-saving-stub guarantee. Modelling the pseudos as
ordinary calls with the standard regmask would also be correct, at some cost to
code quality on targets that do provide the guarantee.
Contributor guide
Assessment
This issue has not been assessed yet.