pypa / pypa/setuptools

Runtime libraries conflict

Open
#916 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

This is a quite general issue and I'm not sure that it relates to the setuptools directly, but there should be some starting point, and I don't know what is the best place to raise this question.

Anyway, here's my environment: FreeBSD 11/Python 3.5/setuptools 32.3.1
There are some math libraries that have been installed from ports: blas/openblas/lapack/atlas. All these libs have been built with gcc and they depend on its runtime libs. It looks like this:

# ldd /usr/local/lib/liblapack.so.4
/usr/local/lib/liblapack.so.4:
	libgfortran.so.3 => /usr/local/lib/gcc49/libgfortran.so.3 (0x801a00000)
	libm.so.5 => /lib/libm.so.5 (0x801e65000)
	libgcc_s.so.1 => /usr/local/lib/gcc49/libgcc_s.so.1 (0x80208f000)
	libquadmath.so.0 => /usr/local/lib/gcc49/libquadmath.so.0 (0x8022a6000)

# ldd /usr/local/lib/gcc49/libgfortran.so.3
/usr/local/lib/gcc49/libgfortran.so.3:
	libquadmath.so.0 => /usr/local/lib/gcc49/libquadmath.so.0 (0x801465000)
	libm.so.5 => /lib/libm.so.5 (0x8016b3000)
	libgcc_s.so.1 => /usr/local/lib/gcc49/libgcc_s.so.1 (0x8018dd000)
	libc.so.7 => /lib/libc.so.7 (0x800823000)

Pay attention for the path of the libgcc_s.so.1 library.

Then, there are some python packages have been installed with pip, like: scipy, scikit-learn and so on. And here is an interesting point. FreeBSD uses Clang as a system compiler, and setuptools/distutils uses the system compiler to build some system-dependent parts of the packages. So, these packages have been build with different compiler and they depend on different set of runtime libs. At the same time these packages depend on the libs from ports. It looks like this:

# ldd /usr/local/lib/python3.5/site-packages/scipy/sparse/linalg/isolve/_iterative.so
/usr/local/lib/python3.5/site-packages/scipy/sparse/linalg/isolve/_iterative.so:
	liblapack.so.4 => /usr/local/lib/liblapack.so.4 (0x801400000)
	libcblas.so.2 => /usr/local/lib/libcblas.so.2 (0x801c34000)
	libblas.so.2 => /usr/local/lib/libblas.so.2 (0x801e51000)
	libpython3.5m.so.1.0 => /usr/local/lib/libpython3.5m.so.1.0 (0x802200000)
	libgfortran.so.3 => /usr/local/lib/gcc49/libgfortran.so.3 (0x802800000)
	libm.so.5 => /lib/libm.so.5 (0x802c65000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802e8f000)
	libquadmath.so.0 => /usr/local/lib/gcc49/libquadmath.so.0 (0x80309d000)
	libc.so.7 => /lib/libc.so.7 (0x800823000)
	libatlas.so.2 => /usr/local/lib/libatlas.so.2 (0x803400000)
	libthr.so.3 => /lib/libthr.so.3 (0x803977000)
	libintl.so.8 => /usr/local/lib/libintl.so.8 (0x803b9e000)
	libutil.so.9 => /lib/libutil.so.9 (0x803da9000)

As a result, the runtime libraries conflict happens when one of the packages is imported:

Traceback (most recent call last):
  File "./***py", line 13, in <module>
    from sklearn import preprocessing
  File "/usr/local/lib/python3.5/site-packages/sklearn/__init__.py", line 57, in <module>
    from .base import clone
  File "/usr/local/lib/python3.5/site-packages/sklearn/base.py", line 12, in <module>
    from .utils.fixes import signature
  File "/usr/local/lib/python3.5/site-packages/sklearn/utils/__init__.py", line 11, in <module>
    from .validation import (as_float_array,
  File "/usr/local/lib/python3.5/site-packages/sklearn/utils/validation.py", line 18, in <module>
    from ..utils.fixes import signature
  File "/usr/local/lib/python3.5/site-packages/sklearn/utils/fixes.py", line 291, in <module>
    from scipy.sparse.linalg import lsqr as sparse_lsqr
  File "/usr/local/lib/python3.5/site-packages/scipy/sparse/linalg/__init__.py", line 112, in <module>
    from .isolve import *
  File "/usr/local/lib/python3.5/site-packages/scipy/sparse/linalg/isolve/__init__.py", line 6, in <module>
    from .iterative import *
  File "/usr/local/lib/python3.5/site-packages/scipy/sparse/linalg/isolve/iterative.py", line 7, in <module>
    from . import _iterative
ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc49/libgfortran.so.3 not found

I've heard there are some workarounds for this issue, but I think there should be more elegant solution.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the scipy/sklearn import on FreeBSD 11 with Python 3.5 and inspect setuptools/distutils compiler and runtime-library handling alongside the ldd output. Define done as an agreed, tested way to prevent or resolve the mixed libgcc_s/libgfortran conflict; the issue does not name a source file or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.