avast / avast/retdec

x86-64 MOV r64, m64 reload is lifted as sign-extended low i32 pointer

Open
#1,249 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Summary

RetDec v5.0 lifts a 64-bit x86-64 pointer reload in sqlite's `balance`
function as a sign-extension of the low 32 bits. The truncated value is then
used as a pointer base and passed to `decodeFlags`.

The machine instruction is:

```asm
9658b: 48 8b 9d 58 fe ff ff movq -424(%rbp), %rbx
```

This is a REX.W 64-bit load. The value loaded into `rbx` is used immediately:

```asm
96592: 4c 8b 63 48 movq 72(%rbx), %r12
96596: 4c 8b 7b 50 movq 80(%rbx), %r15
...
965fb: 48 89 df movq %rbx, %rdi
965fe: e8 ed bf fd ff callq 0x725f0
```

### Reproducer

Input binary and observed output are available here:

https://github.com/x14ngch3n/retdec/tree/poc-issue-1249/poc/issue-1249

Files:

- `sqlite3.debug.xz`: x86-64 PIE sqlite binary with debug info.
- uncompressed SHA-256:
`1f9dbde8d6d6f445f31169b391e8a9994f359ed00110efc79ad4ba6af6d699f0`
- compressed SHA-256:
`ff6cde7166d5a5d4a5a58903aceace345f14b947384cc0a5d9aa0929ecb146ef`
- `retdec.up.ll.xz`: RetDec v5.0 output, disassembled/upgraded to textual
LLVM IR for easier inspection.
- compressed SHA-256:
`75eb6c1d231acd39e3de92914b954c9fd573f3e36faf80693f528b14416bdc93`
- `balance-0x96560-0x96610.objdump.txt`: disassembly window around the
problematic instruction.

Commands:

```sh
xz -d -k sqlite3.debug.xz
retdec-decompiler -s -k -o sqlite3.c sqlite3.debug
```

Then inspect RetDec's LLVM output for function `balance`, around the code
corresponding to machine address `0x9658b`.

### Expected behavior

The lifted value used as `rbx` should preserve the full 64-bit value loaded by:

```asm
movq -424(%rbp), %rbx
```

### Observed behavior

The observed RetDec IR does not contain a literal `trunc i32`, but the
`mul i64 %1597, 4294967296` followed by `ashr exact i64 ..., 32` is equivalent
to taking the low 32 bits and sign-extending them back to i64:

```llvm
%1596 = load ptr, ptr %stack_var_-432, align 8, !asm !60166
%1597 = ptrtoint ptr %1596 to i64
%sext50 = mul i64 %1597, 4294967296
%1598 = ashr exact i64 %sext50, 32, !asm !60166
```

For example, for an i64 value `x`, this computes:

```c
(int64_t)(int32_t)(x & 0xffffffff)
```

That `%1598` value is then used as the pointer base and as the first argument
to `decodeFlags`:

```llvm
%1599 = add nsw i64 %1598, 72, !asm !60167
%1602 = add nsw i64 %1598, 80, !asm !60168
...
%1644 = call i64 @decodeFlags(i64 %1598, i8 %1570), !asm !60187
```

For any valid pointer whose upper 32 bits are non-zero, this does not preserve
the semantics of the machine instruction.

### Patch

I opened a separate PR for a candidate fix and regression test:

https://github.com/avast/retdec/pull/1250

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with poc/issue-1249/sqlite3.debug.xz and the provided retdec-decompiler command, then inspect balance around machine address 0x9658b in retdec.up.ll.xz. Compare the lifted value with the movq instruction and run the regression test from PR #1250; done means the full 64-bit loaded value is preserved for the pointer uses and decodeFlags call.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.