avast / avast/retdec

Decompilation takes too much memory

Open
#13 38 comments 6 reactions 0 assignees View on GitHub
bug C-bin2llvmir C-llvmir2hll P-run T-memory
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

The file I'm decompiling is kinda publicly available but I'm not sure how to share it with devs because the people who have written the code probably don't want it to be decompiled in the first place.

I'd be happy to send the file to a dev via mail or some other means.

I'm running: ./decompiler.sh
Which at some point runs: `llvmir2hll -target-hll=c -var-renamer=readable -var-name-gen=fruit -var-name-gen-prefix= -call-info-obtainer=optim -arithm-expr-evaluator=c -validate-module -llvmir2bir-converter=orig -o [...].dll.c [...].dll.c.backend.bc -enable-debug -emit-debug-comments -config-path=[...].dll.c.json`

Once it reaches the step below, it slowly starts eating memory until at one point it quickly goes from 8GB -> 32 GB and then get's killed by docker.

How would I help debugging this? My guess would be it's running into some infinite depth loop.

```
Running phase: optimizations [normal] ( 52.09s )
-> running RemoveUselessCastsOptimizer ( 52.09s )
-> running UnusedGlobalVarOptimizer ( 52.63s )
-> running DeadLocalAssignOptimizer ( 53.80s )
-> running SimpleCopyPropagationOptimizer ( 65.16s )
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v1_1002017b = (v3_10020166 + 1)` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v1_1002031f = (v3_1002030a + 1)` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v1_100200c9 = (v3_100200b4 + 1)` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v1_10020f22 = (v3_10020f0d + 1)` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `goto 0x10020eca` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `eax_global_2d_to_2d_local = v3_10021abe` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `eax_global_2d_to_2d_local = v3_100224d9` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v10_1002391d = v2_1002386d` -> skipping this edge
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the decompiler.sh command and the llvmir2hll invocation in the report, focusing on the optimizations phase where memory growth begins. Reproduce the run with a shareable input if available, inspect the SimpleCopyPropagationOptimizer and NonRecursiveCFGBuilder warnings, and consider the issue resolved when the same input no longer exhausts Docker's memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker
Domain
reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.