microsoft / microsoft/TRELLIS

how to resolve Command '['ninja']' returned non-zero exit status 1. error

Open
#254 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.7k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

[SPARSE] Backend: spconv, Attention: flash_attn
/usr/local/lib/python3.11/dist-packages/gradio_client/utils.py:1097: UserWarning: file() is deprecated and will be removed in a future version. Use handle_file() instead.
  warnings.warn(
[SPARSE][CONV] spconv algo: native
[ATTENTION] Using backend: flash_attn
[1/40] [GCC][c++/pch]/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h.gch
FAILED: /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h.gch 
g++ -MMD -MT /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h.gch -MF /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h.gch.d -I "/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include" -I "/usr/local/lib/python3.11/dist-packages/pybind11/include" -I "/usr/include/python3.11" -I "/usr/local/cuda/include" -I "/usr/local/lib/python3.11/dist-packages/include" -O3 -DTV_STATIC_VARIABLE_IMPLEMENTATION -std=c++14 -fPIC -x c++-header -DTV_ENABLE_HARDWARE_ACC -c /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h -o /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h.gch
/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h:1:9: warning: #pragma once in main file
    1 | #pragma once
      |         ^~~~
/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h:2:10: fatal error: tensorview/pybind_utils.h: No such file or directory
    2 | #include <tensorview/pybind_utils.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[2/40] [GCC][c++/pch]/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h.gch
FAILED: /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h.gch 
g++ -MMD -MT /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h.gch -MF /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h.gch.d -I "/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include" -I "/usr/local/lib/python3.11/dist-packages/pybind11/include" -I "/usr/include/python3.11" -I "/usr/local/lib/python3.11/dist-packages/include" -I "/usr/local/cuda/include" -O3 -DTV_STATIC_VARIABLE_IMPLEMENTATION -std=c++14 -fPIC -x c++-header -DTV_ENABLE_HARDWARE_ACC -c /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h -o /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h.gch
/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h:1:9: warning: #pragma once in main file
    1 | #pragma once
      |         ^~~~
In file included from /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/csrc/arrayref/ArrayPtr.h:3:
/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/common/TensorViewCPU.h:3:10: fatal error: tensorview/core/all.h: No such file or directory
    3 | #include <tensorview/core/all.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[3/40] [GCC][c++]/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc.o
FAILED: /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc.o 
g++ -MMD -MT /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc.o -MF /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc.o.d -I "/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include" -I "/usr/local/lib/python3.11/dist-packages/pybind11/include" -I "/usr/include/python3.11" -I "/usr/local/cuda/include" -I "/usr/local/lib/python3.11/dist-packages/include" -O3 -DTV_STATIC_VARIABLE_IMPLEMENTATION -std=c++14 -fPIC -DTV_ENABLE_HARDWARE_ACC -c /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc -o /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc.o
In file included from /usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/src/tensorview_bind/PyBindTensorViewBind/PyBindTensorViewBind_bind_TensorViewBind.cc:2:
/usr/local/lib/python3.11/dist-packages/cumm/build/core_cc/include/tensorview_bind/TensorViewBind.h:2:10: fatal error: tensorview/pybind_utils.h: No such file or directory
    2 | #include <tensorview/pybind_utils.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/content/TRELLIS-3D/app.py", line 408, in <module>
    pipeline = TrellisImageTo3DPipeline.from_pretrained("JeffreyXiang/TRELLIS-image-large")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/TRELLIS-3D/trellis/pipelines/trellis_image_to_3d.py", line 56, in from_pretrained
    pipeline = super(TrellisImageTo3DPipeline, TrellisImageTo3DPipeline).from_pretrained(path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/TRELLIS-3D/trellis/pipelines/base.py", line 39, in from_pretrained
    _models = {
              ^
  File "/content/TRELLIS-3D/trellis/pipelines/base.py", line 40, in <dictcomp>
    k: models.from_pretrained(f"{path}/{v}")
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/TRELLIS-3D/trellis/models/__init__.py", line 59, in from_pretrained
    model = __getattr__(config['name'])(**config['args'], **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/TRELLIS-3D/trellis/models/structured_latent_vae/decoder_mesh.py", line 108, in __init__
    SparseSubdivideBlock3d(
  File "/content/TRELLIS-3D/trellis/models/structured_latent_vae/decoder_mesh.py", line 43, in __init__
    sp.SparseConv3d(channels, self.out_channels, 3, indice_key=f"res_{self.out_resolution}"),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/TRELLIS-3D/trellis/modules/sparse/conv/conv_spconv.py", line 11, in __init__
    import spconv.pytorch as spconv
  File "/usr/local/lib/python3.11/dist-packages/spconv/__init__.py", line 15, in <module>
    from . import build as _build
  File "/usr/local/lib/python3.11/dist-packages/spconv/build.py", line 21, in <module>
    from .constants import PACKAGE_NAME, PACKAGE_ROOT, DISABLE_JIT, SPCONV_INT8_DEBUG
  File "/usr/local/lib/python3.11/dist-packages/spconv/constants.py", line 19, in <module>
    from cumm.gemm.constants import NVRTCMode
  File "/usr/local/lib/python3.11/dist-packages/cumm/__init__.py", line 33, in <module>
    pccm.builder.build_pybind([ArrayPtr(), TensorViewBind(), AppleMetalImpl()],
  File "/usr/local/lib/python3.11/dist-packages/pccm/builder/pybind.py", line 121, in build_pybind
    res = ccimport.ccimport(
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/ccimport/core.py", line 155, in ccimport
    target_filename, no_work = build_simple_ninja(
                               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/ccimport/buildtools/writer.py", line 1109, in build_simple_ninja
    raise subprocess.CalledProcessError(proc.returncode, cmds)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.

cuda version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Jun__6_02:18:23_PDT_2024
Cuda compilation tools, release 12.5, V12.5.82
Build cuda_12.5.r12.5/compiler.34385749_0

=== Default Python & Pip Version ===
/usr/local/bin/python
Python 3.11.12
pip 25.1 from /usr/local/lib/python3.11/dist-packages/pip (python 3.11)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with app.py and follow the import chain through trellis/modules/sparse/conv/conv_spconv.py into spconv and cumm. Inspect why the cumm build cannot find tensorview/pybind_utils.h and tensorview/core/all.h in the reported Python 3.11 environment. Done means the pipeline import completes without the ninja failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.