X87ReducedPrecision handles invalid FIST* conversions incorrectly
- Dominant language
- C++
- Stars
- 8k
- Forks
- 351
- Avg merge
- 12h 31m
- Merged PRs (30d)
- 102
Description
With `X87ReducedPrecision=1`, invalid FIST conversions are handled incorrectly. On overflow, NaN, or infinity, it should store the x87 integer indefinite value and set the invalid operation bit. Instead it stores the Arm conversion result and leaves the invalid operation bit clear.
### Reproducer
```nasm
BITS 32
GLOBAL _start
SECTION .data
value: dq 32769.0
result: dw 0
SECTION .text
_start:
finit
fld qword [value]
fistp word [result]
fnstsw ax
mov ebx, 1
test ax, 1
jz .exit
cmp word [result], 0x8000
jne .exit
xor ebx, ebx
.exit:
mov eax, 1
int 0x80
```
```sh
nasm -f elf32 fist.asm -o fist.o
ld -m elf_i386 fist.o -o fist
FEX_X87REDUCEDPRECISION=0 ./fist; echo $? # 0
FEX_X87REDUCEDPRECISION=1 ./fist; echo $? # 1
```
I tracked this down as the root-cause for audio crackling issues in Need for Speed: Most Wanted.
Related: #4729, #5786
Note that #4729 seems insufficient, it only sets the invalid operation bit but not the correct integer indefinite value.
Contributor guide
Research direction
Start with the FIST* conversion path selected by X87ReducedPrecision and use the provided fist.asm reproducer, built with nasm and ld as shown. Compare behavior with FEX_X87REDUCEDPRECISION=0 and 1; done means overflow, NaN, and infinity store the x87 integer indefinite value and set the invalid operation bit, including the case discussed in #4729.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100