llvm / llvm/torch-mlir

Crash while compiling distilGPT2

Open
#1,606 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.9k
Forks
736
Avg merge
5d 22h
Merged PRs (30d)
15

Description

Hi, I'm getting an error while attempting to use torch.compile on the distilGPT2 model from hf. I'm using the following script to compile and run the model: https://gist.github.com/JakopinA/21c61452fce4b046cd58408c673d7117

Here is my crash backtrace:

```
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0. Program arguments: ./build/bin/torch-mlir-opt -pass-pipeline=builtin.module(torchscript-module-to-torch-backend-pipeline) distilGPT2.mlir
#0 0x000056383786897a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:569:11
#1 0x0000563837868b2b PrintStackTraceSignalHandler(void*) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:636:1
#2 0x0000563837867176 llvm::sys::RunSignalHandlers() /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/lib/Support/Signals.cpp:104:5
#3 0x0000563837869255 SignalHandler(int) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
#4 0x00007f30a6667520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
#5 0x00007f30a66bba7c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
#6 0x00007f30a66bba7c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
#7 0x00007f30a66bba7c pthread_kill ./nptl/pthread_kill.c:89:10
#8 0x00007f30a6667476 gsignal ./signal/../sysdeps/posix/raise.c:27:6
#9 0x00007f30a664d7f3 abort ./stdlib/abort.c:81:7
#10 0x00007f30a664d71b _nl_load_domain ./intl/loadmsgcat.c:1177:9
#11 0x00007f30a665ee96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
#12 0x00005638331f9538 decltype(auto) llvm::dyn_cast(mlir::Type const&) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/include/llvm/Support/Casting.h:649:53
#13 0x00005638331f8e25 mlir::IntegerType mlir::Type::dyn_cast() const /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/IR/Types.h:270:10
#14 0x00005638376afc18 mlir::Type::isInteger(unsigned int) const /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/IR/Types.cpp:33:20
#15 0x00005638332744cb (anonymous namespace)::TypeAnalysis::visitOperation(mlir::Operation*, llvm::ArrayRef const*>, llvm::ArrayRef*>) /home/alec/Nod/torch-mlir/lib/Dialect/Torch/Transforms/RefineTypes.cpp:950:9
#16 0x0000563833272dba mlir::dataflow::SparseDataFlowAnalysis>::visitOperationImpl(mlir::Operation*, llvm::ArrayRef, llvm::ArrayRef) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/Analysis/DataFlow/SparseAnalysis.h:276:3
#17 0x00005638374a8eed mlir::dataflow::AbstractSparseDataFlowAnalysis::visitOperation(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp:125:1
#18 0x00005638374a86e7 mlir::dataflow::AbstractSparseDataFlowAnalysis::initializeRecursively(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp:55:25
#19 0x00005638374a8807 mlir::dataflow::AbstractSparseDataFlowAnalysis::initializeRecursively(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp:60:20
#20 0x00005638374a862d mlir::dataflow::AbstractSparseDataFlowAnalysis::initialize(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp:47:10
#21 0x00005638374865c6 mlir::DataFlowSolver::initializeAndRun(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Analysis/DataFlowFramework.cpp:70:25
#22 0x00005638332728cb (anonymous namespace)::RefineTypesPass::runOnOperation() /home/alec/Nod/torch-mlir/lib/Dialect/Torch/Transforms/RefineTypes.cpp:1686:23
#23 0x000056383743060a mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:466:21
#24 0x0000563837430c04 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:529:16
#25 0x0000563837436158 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:749:36
#26 0x0000563837435dc9 mlir::LogicalResult mlir::failableParallelForEach<__gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator>>, __gnu_cxx::__normal_iterator>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/include/mlir/IR/Threading.h:46:18
#27 0x0000563837431ee3 mlir::LogicalResult mlir::failableParallelForEach>&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&>(mlir::MLIRContext*, std::vector>&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/include/mlir/IR/Threading.h:92:10
#28 0x0000563837431783 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:759:14
#29 0x00005638374308b7 mlir::detail::OpToOpPassAdaptor::runOnOperation(bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:650:5
#30 0x00005638374305fb mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:463:5
#31 0x0000563837430c04 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:529:16
#32 0x0000563837434ee1 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_6::operator()(mlir::OpPassManager&, mlir::Operation*) const /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:451:12
#33 0x0000563837434c62 mlir::LogicalResult llvm::function_ref::callback_fn(long, mlir::OpPassManager&, mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#34 0x0000563833249031 llvm::function_ref::operator()(mlir::OpPassManager&, mlir::Operation*) const /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#35 0x0000563833248d15 mlir::Pass::runPipeline(mlir::OpPassManager&, mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/Pass/Pass.h:195:12
#36 0x0000563833246317 (anonymous namespace)::LowerToBackendContractPass::runOnOperation() /home/alec/Nod/torch-mlir/lib/Dialect/Torch/Transforms/LowerToBackendContract.cpp:235:18
#37 0x000056383743060a mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:466:21
#38 0x0000563837430c04 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:529:16
#39 0x000056383743259c mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:834:10
#40 0x00005638374324b2 mlir::PassManager::run(mlir::Operation*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:814:60
#41 0x0000563832c028f2 performActions(llvm::raw_ostream&, bool, bool, llvm::SourceMgr&, mlir::MLIRContext*, llvm::function_ref, bool, bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:91:17
#42 0x0000563832c02559 processBuffer(llvm::raw_ostream&, std::unique_ptr>, bool, bool, bool, bool, bool, bool, llvm::function_ref, mlir::DialectRegistry&, llvm::ThreadPool*) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:139:12
#43 0x0000563832c02338 mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr>, llvm::function_ref, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0::operator()(std::unique_ptr>, llvm::raw_ostream&) const /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:181:12
#44 0x0000563832c0222d mlir::LogicalResult llvm::function_ref>, llvm::raw_ostream&)>::callback_fn>, llvm::function_ref, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0>(long, std::unique_ptr>, llvm::raw_ostream&) /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#45 0x00005638376ef359 llvm::function_ref>, llvm::raw_ostream&)>::operator()(std::unique_ptr>, llvm::raw_ostream&) const /home/alec/Nod/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#46 0x00005638376ee935 mlir::splitAndProcessBuffer(std::unique_ptr>, llvm::function_ref>, llvm::raw_ostream&)>, llvm::raw_ostream&, bool, bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Support/ToolUtilities.cpp:28:12
#47 0x0000563832c01283 mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr>, llvm::function_ref, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:186:10
#48 0x0000563832c0141f mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr>, mlir::PassPipelineCLParser const&, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool, bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:209:10
#49 0x0000563832c02037 mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&, bool) /home/alec/Nod/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:306:14
#50 0x0000563832bac247 main /home/alec/Nod/torch-mlir/tools/torch-mlir-opt/torch-mlir-opt.cpp:26:7
#51 0x00007f30a664ed90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#52 0x00007f30a664ee40 call_init ./csu/../csu/libc-start.c:128:20
#53 0x00007f30a664ee40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#54 0x0000563832bac105 _start (./build/bin/torch-mlir-opt+0x1b9105)
Aborted (core dumped)
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported torch-mlir-opt command using the distilGPT2 input from the linked gist, then inspect lib/Dialect/Torch/Transforms/RefineTypes.cpp around TypeAnalysis::visitOperation. Confirm the failing pipeline no longer aborts on this input and add a regression test if the repository's existing tests cover this pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.