avast / avast/retdec

x87: confusing use of float80_t for fbst

Open
#880 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Subject: https://github.com/rfalke/decompiler-subjects/blob/master/from_holdec/i386_x87_fpu/ia32_elf/subject.exe
Output:
```
int32_t FBSTP(void) {
float80_t v1 = __asm_fbstp(-123456789011223344.0L); // 0x804a06b
*(float80_t *)&g2 = v1;
unsigned char v2 = *(char *)&g7; // 0x804a091
unsigned char v3 = *(char *)&g3; // 0x804a0b1
printf("FBSTP(%f) should=0x44 0x33 0x22 0x11 0x90 0x78 0x56 0x34 0x12 0x80 is=0x%02x 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x\n", -1.2345678901122334e+17, (int32_t)(int80_t)v1 % 256, (int32_t)v3, (int32_t)g4, (int32_t)g5, (int32_t)g6, (int32_t)v2, (int32_t)g8, (int32_t)g9, (int32_t)g10, (int32_t)g11);
return 0;
}
```

While the return type of `__asm_fbstp` is 10 bytes (80 bits) long it is not a floating point type. Therefor the type of v1 should be something else.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the FBSTP output in issue 880 and the __asm_fbstp entry point shown there. Trace how its 10-byte return value becomes float80_t, then verify that the generated output uses a non-floating 80-bit representation while preserving the expected bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.