m-bain / m-bain/webvid

Issues while installing mpi4py on linux based cluster

Open
#17 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
686
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hi, I am trying to run a pip install mpi4py but it throws off with an error. The exact error is here:

```
Defaulting to user installation because normal site-packages is not writeable
Collecting mpi4py
Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 18.3 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
Building wheel for mpi4py (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for mpi4py (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [156 lines of output]
running bdist_wheel
running build
running build_src
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
creating build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
creating build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
creating build/lib.linux-x86_64-cpython-39/mpi4py/include
creating build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
running build_clib
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler: /share/apps/anaconda3-2022.05/bin/mpicc
MPI C++ compiler: /share/apps/anaconda3-2022.05/bin/mpicxx
MPI F compiler: /share/apps/anaconda3-2022.05/bin/mpifort
MPI F90 compiler: /share/apps/anaconda3-2022.05/bin/mpif90
MPI F77 compiler: /share/apps/anaconda3-2022.05/bin/mpif77
checking for library 'lmpe' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
building 'mpe' dylib library
creating build/temp.linux-x86_64-cpython-39
creating build/temp.linux-x86_64-cpython-39/src
creating build/temp.linux-x86_64-cpython-39/src/lib-pmpi
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/mpe.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

warning: build_clib: building optional library "mpe" failed

checking for library 'vt-mpi' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
checking for library 'vt.mpi' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
building 'vt' dylib library
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

warning: build_clib: building optional library "vt" failed

checking for library 'vt-mpi' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
checking for library 'vt.mpi' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
building 'vt-mpi' dylib library
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-mpi.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

warning: build_clib: building optional library "vt-mpi" failed

checking for library 'vt-hyb' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
checking for library 'vt.ompi' ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
building 'vt-hyb' dylib library
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-hyb.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

warning: build_clib: building optional library "vt-hyb" failed

running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler: /share/apps/anaconda3-2022.05/bin/mpicc
MPI C++ compiler: /share/apps/anaconda3-2022.05/bin/mpicxx
MPI F compiler: /share/apps/anaconda3-2022.05/bin/mpifort
MPI F90 compiler: /share/apps/anaconda3-2022.05/bin/mpif90
MPI F77 compiler: /share/apps/anaconda3-2022.05/bin/mpif77
checking for dlopen() availability ...
checking for header 'dlfcn.h' ...
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
success!
removing: _configtest.c _configtest.o
success!
checking for library 'dl' ...
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
checking for function 'dlopen' ...
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
building 'mpi4py.dl' extension
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/share/apps/anaconda3-2022.05/include/python3.9 -c src/dynload.c -o build/temp.linux-x86_64-cpython-39/src/dynload.o
gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -shared -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib build/temp.linux-x86_64-cpython-39/src/dynload.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o build/lib.linux-x86_64-cpython-39/mpi4py/dl.cpython-39-x86_64-linux-gnu.so
checking for MPI compile and link ...
/share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
/share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
failure.
removing: _configtest.c _configtest.o
error: Cannot compile MPI programs. Check your configuration!!!
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects
```
This however works on conda, but even if I can see the package on conda list, it doesn't show up when I try importing it from python. Is there a fix for this? Is this a python version issue? I'm using python 3.10.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported `pip install mpi4py` failure in the Linux cluster environment and inspect the logged `/share/apps/anaconda3-2022.05/bin/mpicc` invocation. Confirm whether the missing `x86_64-conda_cos6-linux-gnu-cc` compiler is an environment issue or an actionable project installation problem; done requires a documented, reproducible installation path or a confirmed external cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.