FEX-Emu / FEX-Emu/FEX

X87ReducedPrecision handles invalid FIST* conversions incorrectly

Open
#5,868 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

With `X87ReducedPrecision=1`, invalid FIST conversions are handled incorrectly. On overflow, NaN, or infinity, it should store the x87 integer indefinite value and set the invalid operation bit. Instead it stores the Arm conversion result and leaves the invalid operation bit clear.

### Reproducer

```nasm
BITS 32
GLOBAL _start

SECTION .data
value: dq 32769.0
result: dw 0

SECTION .text
_start:
finit
fld qword [value]
fistp word [result]
fnstsw ax

mov ebx, 1
test ax, 1
jz .exit
cmp word [result], 0x8000
jne .exit
xor ebx, ebx

.exit:
mov eax, 1
int 0x80
```

```sh
nasm -f elf32 fist.asm -o fist.o
ld -m elf_i386 fist.o -o fist

FEX_X87REDUCEDPRECISION=0 ./fist; echo $? # 0
FEX_X87REDUCEDPRECISION=1 ./fist; echo $? # 1
```

I tracked this down as the root-cause for audio crackling issues in Need for Speed: Most Wanted.

Related: #4729, #5786

Note that #4729 seems insufficient, it only sets the invalid operation bit but not the correct integer indefinite value.

Contributor guide

Open the contributing guide

Research direction

Start with the FIST* conversion path selected by X87ReducedPrecision and use the provided fist.asm reproducer, built with nasm and ld as shown. Compare behavior with FEX_X87REDUCEDPRECISION=0 and 1; done means overflow, NaN, and infinity store the x87 integer indefinite value and set the invalid operation bit, including the case discussed in #4729.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.