[CIR] emitting clang IR for sqlite3 runs out of memory
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
I wanted to test out emitting CIR for the sqlite3 amalgamation.
compiling the sqlite3 amalgamation takes about 2s on my machine through clang, and just dumping the ast took <1s, taking about 400MB of RAM (as expected).
However, if I try to emit cir with `clang -fclangir -emit-cir sqlite3.c`, it ran for 15 minutes, taking up 20GB of memory on my 32GB machine before I killed it. If i limit memory to something like 5GB it'll bad_alloc almost immediately and crash.
I tried to bisect the amalgamation but didn't really get much information, trying to chop the file in half (~70k lines) and trying to get CIR still. Plugging it up to perf, it looks like mallocing took the most time and memory was steadily increasing.
I tried compiling the individual files. btree/select/os_unix/where/pager/expr all emitted CIR in less than 2s, but vdbe.c had the same issue where it wouldn't emit cir and kept eating up memory before bad_allocing.
According to the stack trace it looks like an infinite loop in printing out alises in `mlir/lib/IR/AsmPrinter.cpp`. I tried a pretty naive fix to set the alias.isPrinted to true, which prevents the infinite memory usage but prints types out of order, so this isn't correct.
Stack trace here:
```
Stack dump:
0. Program arguments: /home/takashi/llvm-project-ref/build-cir/bin/clang -fclangir -emit-cir -x c -std=gnu23 -S -o /dev/null /tmp/vdbe_449.i
1. parser at end of file
#0 0x0000561f848e757c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x530b57c)
#1 0x0000561f848e45b5 llvm::sys::RunSignalHandlers() (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x53085b5)
#2 0x0000561f848509b0 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
#3 0x00007f86744447b0 (/usr/lib/libc.so.6+0x447b0)
#4 0x00007f86744ae3cb __pthread_kill_implementation /usr/src/debug/glibc/glibc/nptl/pthread_kill.c:44:76
#5 0x00007f86744ae3cb __pthread_kill_internal /usr/src/debug/glibc/glibc/nptl/pthread_kill.c:89:10
#6 0x00007f86744ae3cb pthread_kill /usr/src/debug/glibc/glibc/nptl/pthread_kill.c:100:10
#7 0x00007f8674444688 raise /usr/src/debug/glibc/glibc/signal/../sysdeps/posix/raise.c:27:6
#8 0x00007f8674425788 internal_signal_block_all /usr/src/debug/glibc/glibc/stdlib/../sysdeps/unix/sysv/linux/internal-signals.h:81:3
#9 0x00007f8674425788 __abort_lock_wrlock /usr/src/debug/glibc/glibc/stdlib/abort.c:62:3
#10 0x00007f8674425788 abort /usr/src/debug/glibc/glibc/stdlib/abort.c:85:3
#11 0x0000561f848568a5 llvm::report_bad_alloc_error(char const*, bool) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x527a8a5)
#12 0x0000561f848568e2 (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x527a8e2)
#13 0x00007f86748ba7d5 operator new(unsigned long) (/usr/lib/libstdc++.so.6+0xba7d5)
#14 0x00007f86749a3eaa std::__cxx11::basic_string, std::allocator>::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (/usr/lib/libstdc++.so.6+0x1a3eaa)
#15 0x00007f86749a5a07 std::__cxx11::basic_string, std::allocator>::_M_append(char const*, unsigned long) (/usr/lib/libstdc++.so.6+0x1a5a07)
#16 0x0000561f848c5d72 llvm::raw_ostream::write(char const*, unsigned long) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x52e9d72)
#17 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#18 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#19 0x0000561f86e698a9 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d8a9)
#20 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#21 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#22 0x0000561f86e6375f cir::PointerType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788775f)
#23 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#24 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#25 0x0000561f86e698a9 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d8a9)
#26 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#27 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#28 0x0000561f86e6375f cir::PointerType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788775f)
#29 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#30 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#31 0x0000561f86e6acbd cir::UnionType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ecbd)
#32 0x0000561f86e68a74 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca74)
#33 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#34 0x0000561f86e698a9 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d8a9)
#35 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#36 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#37 0x0000561f86e6375f cir::PointerType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788775f)
#38 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#39 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#40 0x0000561f86e698a9 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d8a9)
#41 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#42 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#43 0x0000561f86e6375f cir::PointerType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788775f)
#44 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#45 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#46 0x0000561f86e698a9 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d8a9)
#47 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#48 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#49 0x0000561f86e6375f cir::PointerType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788775f)
#50 0x0000561f86e68726 cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788c726)
#51 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#52 0x0000561f86e697f7 cir::StructType::print(mlir::AsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788d7f7)
#53 0x0000561f86e68a0a cir::CIRDialect::printType(mlir::Type, mlir::DialectAsmPrinter&) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x788ca0a)
#54 0x0000561f87de25a1 mlir::AsmPrinter::Impl::printDialectType(mlir::Type) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88065a1)
#55 0x0000561f87df2262 (anonymous namespace)::AliasState::printAliases(mlir::AsmPrinter::Impl&, (anonymous namespace)::NewLineCounter&, bool) AsmPrinter.cpp:0:0
#56 0x0000561f87de53c0 mlir::Operation::print(llvm::raw_ostream&, mlir::AsmState&) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x88093c0)
#57 0x0000561f87de5027 mlir::Operation::print(llvm::raw_ostream&, mlir::OpPrintingFlags const&) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x8809027)
#58 0x0000561f859b306c cir::CIRGenConsumer::HandleTranslationUnit(clang::ASTContext&) CIRGenAction.cpp:0:0
#59 0x0000561f88334179 clang::ParseAST(clang::Sema&, bool, bool) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x8d58179)
#60 0x0000561f8569374b clang::FrontendAction::Execute() (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x60b774b)
#61 0x0000561f856099cf clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x602d9cf)
#62 0x0000561f8574af79 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x616ef79)
#63 0x0000561f830813c1 cc1_main(llvm::ArrayRef, char const*, void*) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x3aa53c1)
#64 0x0000561f8307db1e ExecuteCC1Tool(llvm::SmallVectorImpl&, llvm::ToolContext const&, llvm::IntrusiveRefCntPtr) driver.cpp:0:0
#65 0x0000561f8307f588 int llvm::function_ref&)>::callback_fn(long, llvm::SmallVectorImpl&) driver.cpp:0:0
#66 0x0000561f85434479 void llvm::function_ref::callback_fn>, std::__cxx11::basic_string, std::allocator>*, bool*) const::$_0>(long) Job.cpp:0:0
#67 0x0000561f84850728 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x5274728)
#68 0x0000561f85433d55 clang::driver::CC1Command::Execute(llvm::ArrayRef>, std::__cxx11::basic_string, std::allocator>*, bool*) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x5e57d55)
#69 0x0000561f853ea1ce clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x5e0e1ce)
#70 0x0000561f853eb538 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl>&, bool) const (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x5e0f538)
#71 0x0000561f8540e3af clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl>&) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x5e323af)
#72 0x0000561f8307d4d2 clang_main(int, char**, llvm::ToolContext const&) (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x3aa14d2)
#73 0x0000561f8308c0cc main (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x3ab00cc)
#74 0x00007f8674427c4e __libc_start_call_main /usr/src/debug/glibc/glibc/csu/../sysdeps/nptl/libc_start_call_main.h:83:3
#75 0x00007f8674427d8b call_init /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:128:20
#76 0x00007f8674427d8b __libc_start_main /usr/src/debug/glibc/glibc/csu/../csu/libc-start.c:347:5
#77 0x0000561f8307b625 _start (/home/takashi/llvm-project-ref/build-cir/bin/clang+0x3a9f625)
clang: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 24.0.0git (https://github.com/Takashiidobe/llvm-project.git 7678fad242ceff7fcd2b3af798123c6d2d7166e3)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/takashi/llvm-project-ref/build-cir/bin
```
[vdbe_449-412200.c](https://github.com/user-attachments/files/31352090/vdbe_449-412200.c)
[vdbe_449-412200.sh](https://github.com/user-attachments/files/31352089/vdbe_449-412200.sh)
Contributor guide
Assessment
This issue has not been assessed yet.