avast / avast/retdec

retdec-llvmir2hll fails in SimpleCopyPropagationOptimizer due to insufficient stack space

Open
#403 7 comments 0 reactions 0 assignees View on GitHub
bug C-llvmir2hll P-run T-memory
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Hello!

Decompiling (retdec-llvmir2hll) Teslacrypt (https://github.com/ytisf/theZoo/tree/master/malwares/Binaries/Ransomware.TeslaCrypt)
fails with a huge stacktrace for reasons I currently don't understand:

```
Running phase: alias analysis [simple] ( 8.13s )
Running phase: optimizations [normal] ( 8.41s )
-> running RemoveUselessCastsOptimizer ( 8.41s )
-> running UnusedGlobalVarOptimizer ( 8.65s )
-> running DeadLocalAssignOptimizer ( 8.91s )
-> running SimpleCopyPropagationOptimizer ( 12.34s )
#0 0x000000000096e4c5 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/local/bin/retdec-llvmir2hll+0x96e4c5)
#1 0x000000000096c29e llvm::sys::RunSignalHandlers() (/usr/local/bin/retdec-llvmir2hll+0x96c29e)
#2 0x000000000096c4d6 SignalHandler(int) (/usr/local/bin/retdec-llvmir2hll+0x96c4d6)
#3 0x00007faa50423f70 __restore_rt (/lib64/libpthread.so.0+0x11f70)
#4 0x000000000050ba1a std::pair >, bool> std::_Rb_tree, std::shared_ptr, std::_Identity >, std::less >, std::allocator > >::_M_insert_unique const&>(std::shared_ptr const&) (/usr/local/bin/retdec-llvmir2hll+0x50ba1a)
#5 0x000000000053370a retdec::llvmir2hll::ValueAnalysis::visit(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x53370a)
#6 0x000000000067d5fa retdec::llvmir2hll::Variable::accept(retdec::llvmir2hll::Visitor*) (/usr/local/bin/retdec-llvmir2hll+0x67d5fa)
#7 0x00000000008ab6e7 retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x8ab6e7)
#8 0x000000000053004e retdec::llvmir2hll::ValueAnalysis::visit(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x53004e)
#9 0x00000000005cafea retdec::llvmir2hll::AddOpExpr::accept(retdec::llvmir2hll::Visitor*) (/usr/local/bin/retdec-llvmir2hll+0x5cafea)
#10 0x000000000052e80f retdec::llvmir2hll::ValueAnalysis::visit(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x52e80f)
#11 0x00000000005d2087 retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor*) (/usr/local/bin/retdec-llvmir2hll+0x5d2087)
#12 0x000000000052e659 retdec::llvmir2hll::ValueAnalysis::getValueData(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x52e659)
#13 0x000000000058e9a3 retdec::llvmir2hll::OptimFuncInfoCFGTraversal::updateFuncInfo(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x58e9a3)
#14 0x000000000058f1e2 retdec::llvmir2hll::OptimFuncInfoCFGTraversal::visitStmt(std::shared_ptr) (/usr/local/bin/retdec-llvmir2hll+0x58f1e2)
#15 0x0000000000582308 retdec::llvmir2hll::CFGTraversal::performTraversalImpl(std::shared_ptr, __gnu_cxx::__normal_iterator const*, std::vector, std::allocator > > >) (/usr/local/bin/retdec-llvmir2hll+0x582308)

[...]

#255 0x0000000000582476 retdec::llvmir2hll::CFGTraversal::performTraversalImpl(std::shared_ptr, __gnu_cxx::__normal_iterator const*, std::vector, std::allocator > > >) (/usr/local/bin/retdec-llvmir2hll+0x582476)
Stack dump:
0. Program arguments: /usr/local/bin/retdec-llvmir2hll -target-hll=c -var-renamer=readable -var-name-gen=fruit -var-name-gen-prefix= -call-info-obtainer=optim -arithm-expr-evaluator=c -validate-module -llvmir2bir-converter=orig -o [DIR]/teslacrypt/3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370.c [DIR]/teslacrypt/3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370.c.backend.bc -enable-debug -emit-debug-comments -config-path=[DIR]/teslacrypt/3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370.c.json -max-memory-half-ram
1. Running pass 'Decompiler' on module '[DIR]/teslacrypt/3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370.c.backend.bc'.
Error: Decompilation of file [DIR]/teslacrypt/3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370.c.backend.bc failed
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with retdec-llvmir2hll on the Teslacrypt input using the command shown in the report, focusing on SimpleCopyPropagationOptimizer, ValueAnalysis, and CFGTraversal in the stack trace. Trace the insufficient-stack-space path and verify that the same input completes decompilation without the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.