avast / avast/retdec

Parameters with type float/double not detected on ARM64

Open
#589 0 comments 0 reactions 0 assignees View on GitHub
bug C-bin2llvmir P-output T-arch-arm64
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

When decompiling ARM64 binary containing a function with FP parameter RetDec does not detect these parameters. This is because the specification of ARM64 calling convention defines that FP parameters are passed in subregisters of larger vector registers and parameter analysis expects to find usage of these registers. However, in RetDec are float and double ARM64 registers implemented as separate sets that are not related to each other. On real architecture this sets overlay on one memory location and change to register from one set affects value in register from another set.

In RetDec it is possible to specify double and float registers used to pass parameters but currently, RetDec is unable to specify two (or more) interconnected sets of registers. With the implementation of ARM64 registers if we specified separate double and float sets the result of decompilation would not be correct all the time.

For example following function:
```
float avg(double a float b)
```

Would be decompiled as:
```
float32_t avg(float64_t a1)
```
The problem is in parameter analysis - filtration specifically. RetDec gathers potential parameters for each definition and call of a function. Found values are subsequently eliminated based on different criteria.

Registers are eliminated based on template registers specified in an adequate calling convention of source file architecture. The interface of calling convention specification provides a way to define registers that are used for double parameters and registers that are used for float parameters. During the phase of filtration are this specified registers taken as different registers and their relationship is not taken to account.

**Possible solutions**
1. (the easiest one) Override filtration method that is responsible for eliminating potential parameters. This method will be invoked instead of the general one. This solution was used for x64 Microsoft convention, specifically to eliminate registers from interconnected groups of GPR and XMM registers. This solution was used because it was thought that MS convention is an exception - it starts to be clear that this might be a case for more conventions.

2. (the best one) Provide a way to specify register groups. This groups will be analyzed with respect to each other and the relationship between them will be preserved.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the ARM64 calling-convention register specification and the parameter-analysis filtration phase described in the issue. Compare how float and double register sets are treated and evaluate the two proposed implementation directions. Done means the example function's float and double parameters are both detected correctly during decompilation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.