avast / avast/retdec

retdec-llvmir2hll crashes in the SimpleCopyPropagationOptimizer phase

Open
#144 4 comments 0 reactions 0 assignees View on GitHub
bug C-llvmir2hll P-run T-memory
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Hello,

I'm trying to decompile a rather larger (3.5MB) ELF x86 binary and I am seeing a crash during the optimizations phase of decompilation. I'm hoping you might have some pointers on how to prevent this crash, or a way to work around it (maybe ignore/skip optimizations somehow).

### Issue Description

#### What command am I running?
A pretty vanilla command:
```bash
./retdec-decompiler.sh -o /home/user/mybinary.ext /home/user/mybinary
```
The binary is stripped but not packed.

#### What am I seeing?
The process makes it through this decompile step:
```
##### Decompiling /home/user/mybinary into /home/user/mybinary.ext.backend.bc....
````
Then fails during:
```
##### Decompiling /home/user/mybinary.ext.backend.bc into /home/user/mybinary.ext
````

#### What error message am I seeing?
After making it through several phases of the second Decompiling step, it fails with the following information:
```
-> converting function_82f85c0() ( 138.19s )
-> converting _fini() ( 138.19s )
-> converting function_82f8600() ( 138.19s )
Running phase: removing functions prefixed with [__decompiler_undefined_function_] ( 138.35s )
Running phase: removing functions from standard libraries ( 149.06s )
-> removing tolower() ( 149.06s )
Running phase: removing code that is not reachable in a CFG ( 149.06s )
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v3_82e3b47 = *(IntToPtrCastExpr((v0_82e3b47 + 4)))` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v3_82e4507 = *(IntToPtrCastExpr((v0_82e4507 + 4)))` -> skipping this edge
Running phase: signed/unsigned types fixing ( 153.00s )
Running phase: converting LLVM intrinsic functions to standard functions ( 166.99s )
Running phase: obtaining debug information ( 169.24s )
Running phase: alias analysis [simple] ( 169.41s )
Running phase: optimizations [normal] ( 171.17s )
-> running RemoveUselessCastsOptimizer ( 171.17s )
-> running UnusedGlobalVarOptimizer ( 172.46s )
-> running DeadLocalAssignOptimizer ( 174.33s )
-> running SimpleCopyPropagationOptimizer ( 224.00s )
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v3_82e3b47 = *(IntToPtrCastExpr((v0_82e3b47 + 4)))` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v3_82e4507 = *(IntToPtrCastExpr((v0_82e4507 + 4)))` -> skipping this edge
./retdec-decompiler.sh: line 940: 25960 Killed "$LLVMIR2HLL" "${LLVMIR2HLL_PARAMS[@]}"
Error: Decompilation of file '/home/user/mybinary.ext.backend.bc' failed
```

So something is going wrong around here... https://github.com/avast-tl/retdec/blob/master/scripts/retdec-decompiler.sh#L945

What is interesting is the "Killed" message. Is something getting forcefully killed because it is taking too long maybe?

### My System Information

#### retdec Version
I successfully cloned and built the project yesterday 1/31/2018

#### OS Info
x86_64 Ubuntu 16.04.3 (xenial)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the retdec-decompiler.sh command and the provided ELF context. Inspect scripts/retdec-decompiler.sh around line 945 and trace the SimpleCopyPropagationOptimizer phase using the logged optimizer sequence. Determine whether the process is being resource-killed or the optimizer is failing, then document a reproducible fix or supported way to skip optimizations.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, shell
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.