avast / avast/retdec

Building `bin2pat` and `pat2yara` crashes clang on macOS 10.6

Open
#1,114 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

An attempt to build retdec on macOS 10.6 via MacPorts works well with two exceptions: `pat2yara` and `bin2pat`.

Both tools is crashed clang-11 like this:
```
snow-leopard:bin2pat catap$ /opt/local/bin/clang++-mp-11 -pipe -I/opt/local/libexec/openssl3/include -Os -nostdinc++ -isystem/opt/local/include/libcxx/v1 -isystem/opt/local/include/LegacySupport -stdlib=libc++ -fvisibility-inlines-hidden
-Wall -Wextra -Wcast-align -Wcast-qual -Wswitch-default -Wuninitialized -Wold-style-cast -Wno-unused-parameter -DNDEBUG -arch x86_64 -mmacosx-version-min=10.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/libexec
/openssl3/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/libcxx -L/opt/local/lib -lMacportsLegacySupport CMakeFiles/bin2pat.dir/bin2pat.cpp.o -o retdec-bin2pat -Wl,-rpath,@executable_path/../lib ../patterngen/lib
retdec-patterngen.a ../utils/libretdec-utils.a ../../deps/install/yaramod/lib/libyaramod.a ../../deps/install/yaramod/lib/libpog_fmt.a ../../deps/install/yaramod/lib/libpog_re2.a ../fileformat/libretdec-fileformat.a ../common/libretdec-c
ommon.a ../utils/libretdec-utils.a ../pelib/libretdec-pelib.a ../../deps/install/llvm/lib/libLLVMDebugInfoDWARF.a ../../deps/install/llvm/lib/libLLVMBitWriter.a ../../deps/install/llvm/lib/libLLVMIRReader.a ../../deps/install/llvm/lib/li
bLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/install/llvm/lib/libLL
VMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm/lib/libLLVMScalarOpt
s.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/libLLVMMC.a ../../dep
s/install/llvm/lib/libLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/i
nstall/llvm/lib/libLLVMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm
/lib/libLLVMScalarOpts.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/
libLLVMMC.a ../../deps/install/llvm/lib/libLLVMPasses.a /opt/local/lib/libz.dylib ../../deps/tlsh/libretdec-tlsh.a ../../deps/stb/libstb.a ../../deps/authenticode-parser/libauthenticode.a /opt/local/libexec/openssl3/lib/libcrypto.dylib /
opt/local/libexec/openssl3/lib/libcrypto.dylib
clang: warning: argument unused during compilation: '-nostdinc++' [-Wunused-command-line-argument]
clang: error: unable to execute command: Segmentation fault
clang: error: linker command failed due to signal (use -v to see invocation)
snow-leopard:bin2pat catap$
```
when I run it with `-v` I see that LLVM's ld is crashed:
```
snow-leopard:bin2pat catap$ /opt/local/bin/clang++-mp-11 -pipe -I/opt/local/libexec/openssl3/include -Os -nostdinc++ -isystem/opt/local/include/libcxx/v1 -isystem/opt/local/include/LegacySupport -stdlib=libc++ -fvisibility-inlines-hidden
-Wall -Wextra -Wcast-align -Wcast-qual -Wswitch-default -Wuninitialized -Wold-style-cast -Wno-unused-parameter -DNDEBUG -arch x86_64 -mmacosx-version-min=10.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/libexec
/openssl3/lib -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/libcxx -L/opt/local/lib -lMacportsLegacySupport CMakeFiles/bin2pat.dir/bin2pat.cpp.o -o retdec-bin2pat -Wl,-rpath,@executable_path/../lib ../patterngen/lib
retdec-patterngen.a ../utils/libretdec-utils.a ../../deps/install/yaramod/lib/libyaramod.a ../../deps/install/yaramod/lib/libpog_fmt.a ../../deps/install/yaramod/lib/libpog_re2.a ../fileformat/libretdec-fileformat.a ../common/libretdec-c
ommon.a ../utils/libretdec-utils.a ../pelib/libretdec-pelib.a ../../deps/install/llvm/lib/libLLVMDebugInfoDWARF.a ../../deps/install/llvm/lib/libLLVMBitWriter.a ../../deps/install/llvm/lib/libLLVMIRReader.a ../../deps/install/llvm/lib/li
bLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/install/llvm/lib/libLL
VMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm/lib/libLLVMScalarOpt
s.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/libLLVMMC.a ../../dep
s/install/llvm/lib/libLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/i
nstall/llvm/lib/libLLVMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm
/lib/libLLVMScalarOpts.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/
libLLVMMC.a ../../deps/install/llvm/lib/libLLVMPasses.a /opt/local/lib/libz.dylib ../../deps/tlsh/libretdec-tlsh.a ../../deps/stb/libstb.a ../../deps/authenticode-parser/libauthenticode.a /opt/local/libexec/openssl3/lib/libcrypto.dylib /
opt/local/libexec/openssl3/lib/libcrypto.dylib -v
clang version 11.1.0
Target: x86_64-apple-darwin10.8.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-11/bin
clang: warning: argument unused during compilation: '-nostdinc++' [-Wunused-command-line-argument]
"/opt/local/libexec/llvm-11/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.6.0 -o retdec-bin2pat -lcrt1.10.6.o -L/opt/local/libexec/openssl3/lib -L/opt/local/lib -L/opt/local/lib/libcxx -L/opt/local/lib -search_paths_first -headerpad_max_install_names -headerpad_max_install_names -lMacportsLegacySupport CMakeFiles/bin2pat.dir/bin2pat.cpp.o -rpath @executable_path/../lib ../patterngen/libretdec-patterngen.a ../utils/libretdec-utils.a ../../deps/install/yaramod/lib/libyaramod.a ../../deps/install/yaramod/lib/libpog_fmt.a ../../deps/install/yaramod/lib/libpog_re2.a ../fileformat/libretdec-fileformat.a ../common/libretdec-common.a ../utils/libretdec-utils.a ../pelib/libretdec-pelib.a ../../deps/install/llvm/lib/libLLVMDebugInfoDWARF.a ../../deps/install/llvm/lib/libLLVMBitWriter.a ../../deps/install/llvm/lib/libLLVMIRReader.a ../../deps/install/llvm/lib/libLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/install/llvm/lib/libLLVMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm/lib/libLLVMScalarOpts.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/libLLVMMC.a ../../deps/install/llvm/lib/libLLVMObject.a ../../deps/install/llvm/lib/libLLVMBinaryFormat.a ../../deps/install/llvm/lib/libLLVMInstCombine.a ../../deps/install/llvm/lib/libLLVMSupport.a ../../deps/install/llvm/lib/libLLVMDemangle.a ../../deps/install/llvm/lib/libLLVMipo.a ../../deps/install/llvm/lib/libLLVMAsmParser.a ../../deps/install/llvm/lib/libLLVMBitReader.a ../../deps/install/llvm/lib/libLLVMMCParser.a ../../deps/install/llvm/lib/libLLVMCodeGen.a ../../deps/install/llvm/lib/libLLVMScalarOpts.a ../../deps/install/llvm/lib/libLLVMTransformUtils.a ../../deps/install/llvm/lib/libLLVMAnalysis.a ../../deps/install/llvm/lib/libLLVMTarget.a ../../deps/install/llvm/lib/libLLVMCore.a ../../deps/install/llvm/lib/libLLVMMC.a ../../deps/install/llvm/lib/libLLVMPasses.a /opt/local/lib/libz.dylib ../../deps/tlsh/libretdec-tlsh.a ../../deps/stb/libstb.a ../../deps/authenticode-parser/libauthenticode.a /opt/local/libexec/openssl3/lib/libcrypto.dylib /opt/local/libexec/openssl3/lib/libcrypto.dylib -lc++ -lSystem /opt/local/libexec/llvm-11/lib/clang/11.1.0/lib/darwin/libclang_rt.osx.a
clang: error: unable to execute command: Segmentation fault
clang: error: linker command failed due to signal (use -v to see invocation)
snow-leopard:bin2pat catap$
```
if I run `ld` in `gdb` I see that it crashed on parsing some static libraries:
```
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00007fff5f3999b8
0x00000001000b86e5 in mach_o::relocatable::Parser::parse ()
(gdb) bt
#0 0x00000001000b86e5 in mach_o::relocatable::Parser::parse ()
#1 0x00000001000b8e5f in mach_o::relocatable::Parser::parse ()
#2 0x0000000100089643 in mach_o::relocatable::parse ()
#3 0x000000010006ff07 in archive::File::makeObjectFileForMember ()
#4 0x00000001000709de in archive::File::justInTimeforEachAtom ()
#5 0x000000010000de13 in ld::tool::InputFiles::searchLibraries ()
#6 0x0000000100064785 in ld::tool::Resolver::resolveUndefines ()
#7 0x0000000100066579 in ld::tool::Resolver::resolve ()
#8 0x000000010000b896 in main ()
(gdb)
```
and if I try to ping point a library which cause a crash, I'll find that `../../deps/install/yaramod/lib/libyaramod.a` cause such crash.

Anyway, disabling build of this two subprojects are enough.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the CMake build of bin2pat and pat2yara on macOS 10.6, using the reported clang-11 linker command and the libyaramod.a archive identified in the report. Reproduce the ld crash under gdb and determine whether the two subprojects can build successfully or must be disabled; done means the affected build no longer crashes and the supported workaround is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, macos
Domain
build-system, compilers, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.