avast / avast/retdec

Decompiler failing at optimization using RDA phase

Open
#930 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

```
$ retdec-decompiler $(which ls)
Running phase: Unpacking ( 0.01s )
No matching plugins found for 'GCC 9.2.1'
No matching plugins found for 'GCC 4.7.0'
Running phase: Initialization ( 0.02s )
Running phase: Providers initialization ( 0.02s )
Running phase: Input binary to LLVM IR decoding ( 0.06s )
Running phase: LLVM ( 0.15s )
Running phase: x86 address spaces optimization ( 0.17s )
Running phase: x87 fpu register analysis ( 0.18s )
Running phase: Main function identification optimization ( 0.20s )
Running phase: Libgcc idioms optimization ( 0.20s )
Running phase: LLVM instruction optimization ( 0.20s )
Running phase: Conditional branch optimization ( 0.21s )
Running phase: Syscalls optimization ( 0.29s )
Running phase: Stack optimization ( 0.29s )
Running phase: Constants optimization ( 0.44s )
Running phase: Function parameters and returns optimization ( 0.60s )
Running phase: LLVM instruction optimization using RDA ( 0.75s )
free(): invalid pointer
#0 0x00001301116ef6e1 (retdec-decompiler+0xd6c6e1)
#1 0x00001301116d1dc4 (retdec-decompiler+0xd4edc4)
#2 0x000073c6b5c41960 __restore_rt (/usr/bin/../lib/libpthread.so.0+0x13960)
#3 0x000073c6b573eef5 raise (/usr/bin/../lib/libc.so.6+0x3cef5)
#4 0x000073c6b5728862 abort (/usr/bin/../lib/libc.so.6+0x26862)
#5 0x000073c6b5780f38 __libc_message (/usr/bin/../lib/libc.so.6+0x7ef38)
#6 0x000073c6b5788bea (/usr/bin/../lib/libc.so.6+0x86bea)
#7 0x000073c6b5789fbc _int_free (/usr/bin/../lib/libc.so.6+0x87fbc)
#8 0x000073c6b578dca8 free (/usr/bin/../lib/libc.so.6+0x8bca8)
#9 0x0000130111c88153 (retdec-decompiler+0x1305153)
#10 0x000013011122f9ac (retdec-decompiler+0x8ac9ac)
#11 0x00001301111c0785 (retdec-decompiler+0x83d785)
#12 0x0000130111ec2660 (retdec-decompiler+0x153f660)
#13 0x000013011115ac45 (retdec-decompiler+0x7d7c45)
#14 0x0000130110f83e73 (retdec-decompiler+0x600e73)
#15 0x000073c6b5729b25 __libc_start_main (/usr/bin/../lib/libc.so.6+0x27b25)
#16 0x000013011113b6fe (retdec-decompiler+0x7b86fe)
Stack dump:
0. Program arguments: retdec-decompiler /bin/ls
1. Running pass 'LLVM instruction optimization using RDA' on module ''.
Aborted (core dumped)
```
and
```
$ retdec-decompiler $(which ls)
Running phase: Unpacking ( 0.01s )
No matching plugins found for 'GCC 9.2.1'
No matching plugins found for 'GCC 4.7.0'
Running phase: Initialization ( 0.02s )
Running phase: Providers initialization ( 0.03s )
Running phase: Input binary to LLVM IR decoding ( 0.07s )
Running phase: LLVM ( 0.21s )
Running phase: x86 address spaces optimization ( 0.23s )
Running phase: x87 fpu register analysis ( 0.26s )
Running phase: Main function identification optimization ( 0.28s )
Running phase: Libgcc idioms optimization ( 0.28s )
Running phase: LLVM instruction optimization ( 0.28s )
Running phase: Conditional branch optimization ( 0.31s )
Running phase: Syscalls optimization ( 0.41s )
Running phase: Stack optimization ( 0.41s )
Running phase: Constants optimization ( 0.55s )
Running phase: Function parameters and returns optimization ( 0.73s )
Running phase: LLVM instruction optimization using RDA ( 0.88s )
#0 0x0000148c925fd6e1 (retdec-decompiler+0xd6c6e1)
#1 0x0000148c925dfdc4 (retdec-decompiler+0xd4edc4)
#2 0x00007355942a5960 __restore_rt (/usr/bin/../lib/libpthread.so.0+0x13960)
#3 0x0000148c9213d8d4 (retdec-decompiler+0x8ac8d4)
#4 0x0000148c920ce785 (retdec-decompiler+0x83d785)
#5 0x0000148c92dd0660 (retdec-decompiler+0x153f660)
#6 0x0000148c92068c45 (retdec-decompiler+0x7d7c45)
#7 0x0000148c91e91e73 (retdec-decompiler+0x600e73)
#8 0x0000735593d8db25 __libc_start_main (/usr/bin/../lib/libc.so.6+0x27b25)
#9 0x0000148c920496fe (retdec-decompiler+0x7b86fe)
Stack dump:
0. Program arguments: retdec-decompiler /bin/ls
1. Running pass 'LLVM instruction optimization using RDA' on module ''.
Segmentation fault (core dumped)
```

the host system is Arch Linux, package successfully compiled from the AUR using current master branch - https://aur.archlinux.org/packages/retdec-git/

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with `retdec-decompiler $(which ls)` on the reported Arch Linux setup and observe the `LLVM instruction optimization using RDA` phase. Trace that RDA optimization entry point to determine why it aborts, then verify that decompilation proceeds past the phase without an invalid free or segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.