llvm / llvm/torch-mlir

`ninja all install` does not install _jit_ir_importer binaries/packages

Open
#2,121 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.9k
Forks
736
Avg merge
5d 22h
Merged PRs (30d)
15

Description

Hi, after building, installing, and exporting libs with:

export PYTHONPATH=:/opt/llvm-project/python_packages/mlir_core:/installs/torch-mlir/python_packages/torch_mlir

I get an error stating that: ModuleNotFoundError: No module named 'torch_mlir._mlir_libs._jit_ir_importer'

If I export the build folder as suggested in the Development.md file, I have no issues.

export PYTHONPATH=:/opt/llvm-project/python_packages/mlir_core:/builds/torch-mlir/tools/torch-mlir/python_packages/torch_mlir

ninja check-torch-mlir-all passes tests (if on green checkmark commit).

One of the missing files that fails to install is here:

/builds/torch-mlir/tools/torch-mlir/python_packages/torch_mlir/torch_mlir/_mlir_libs/_jit_ir_importer.cpython-38-x86_64-linux-gnu.so

How can I install the missing libraries?

My actual project depends on an older version of torch-mlir which does not have published torch-mlir python whees. Directions on generating wheels that include the missing libraries would also be appreciated.


Commands to reproduce:

# clone torch-mlir, initialize submodules and compile with...

mkdir -p /builds/torch-mlir /installs/torch-mlir && \
cmake -GNinja \
    -B/builds/torch-mlir \
    -H/torch_mlir_src/externals/llvm-project/llvm \
    -DCMAKE_INSTALL_PREFIX=/installs/torch-mlir \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_C_COMPILER_LAUNCHER=ccache \
    -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
    -DLLVM_ENABLE_PROJECTS=mlir \
    -DLLVM_EXTERNAL_PROJECTS="torch-mlir;torch-mlir-dialects" \
    -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR=/torch_mlir_src \
    -DLLVM_EXTERNAL_TORCH_MLIR_DIALECTS_SOURCE_DIR=/torch_mlir_src/externals/llvm-external-projects/torch-mlir-dialects \
    -DMLIR_ENABLE_BINDINGS_PYTHON=ON \
    -DLLVM_ENABLE_ASSERTIONS=ON \
    -DLLVM_TARGETS_TO_BUILD=host \
    -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=1 && \
cd /builds/torch-mlir && \
ninja tools/torch-mlir/all install

Added -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=1 just to make sure it is on. It seems to be default to be on.

python -c "import sys; print('\n'.join(sys.path))"
/builds/torch-mlir
/opt/llvm-project/python_packages/mlir_core
/installs/torch-mlir/python_packages/torch_mlir
/usr/lib/python38.zip
/usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/usr/local/lib/python3.8/dist-packages
/usr/lib/python3/dist-packages
(reverse-i-search)`-c': python ^C "import sys; print('\n'.join(sys.path))"
python -c "import torch_mlir"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/installs/torch-mlir/python_packages/torch_mlir/torch_mlir/__init__.py", line 18, in <module>
    from torch_mlir.dialects.torch.importer.jit_ir import ClassAnnotator, ImportOptions, ModuleBuilder
  File "/installs/torch-mlir/python_packages/torch_mlir/torch_mlir/dialects/torch/importer/jit_ir/__init__.py", line 14, in <module>
    from ....._mlir_libs._jit_ir_importer import *
ModuleNotFoundError: No module named 'torch_mlir._mlir_libs._jit_ir_importer'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/development.md and the ninja tools/torch-mlir/all install command, comparing the build and install package paths shown in the report. Trace how _jit_ir_importer.cpython-38-x86_64-linux-gnu.so is handled, then verify that python -c "import torch_mlir" succeeds from the installed path and that generated wheels include the binary.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, python, pytorch
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.