pypa / pypa/cibuildwheel

Missing library files at repair step & wheel configuration through pyproject

Open
#1,981 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.3k
Forks
326
Avg merge
1d 18h
Merged PRs (30d)
8

Description

Description

First of all, thank you for providing cibw to the community!

Missing library files at repair step

I'm trying to provide a pre-built linux only wheel for OpenBabel. I'm taking inspiration from https://github.com/njzjz/openbabel-wheel which is unfortunately not maintained anymore.

The basic setup is that the repository I build only has a CMakeLists.txt that will be used by cibw/scikit-build-core to build the wheel:

CMakeLists.txt 
cmake_minimum_required(VERSION 3.11)
project(ObabelPythonDistributions CXX)

# download OpenBabel to ${OBABEL_SOURCE_ROOT}
include(FetchContent)
FetchContent_Declare(openbabel
  GIT_REPOSITORY    https://github.com/openbabel/openbabel
  GIT_TAG           ${OBABEL_VERSION}
)
FetchContent_GetProperties(openbabel)
if(NOT openbabel_POPULATED)
  FetchContent_Populate(openbabel)
  add_subdirectory(${openbabel_SOURCE_DIR} ${openbabel_BINARY_DIR})
  install(
    DIRECTORY ${openbabel_SOURCE_DIR}/scripts/python/openbabel
    DESTINATION ${CMAKE_INSTALL_PREFIX}/openbabel
  )
endif()

My pyproject defines the version and build flags I want to use:

pyproject.toml 
[tool.scikit-build]
minimum-version = "0.3.0"

[tool.scikit-build.cmake.define]
OBABEL_VERSION = "f495cc6"
CMAKE_BUILD_TYPE = "Release"
WITH_INCHI = "ON"
PYTHON_BINDINGS = "ON"
RUN_SWIG = "ON"

[tool.cibuildwheel]
archs = ["x86_64"]
build = ["cp311-manylinux2014-x86_64"]
test-command = [
    'python -c "from openbabel import openbabel"',
    "obabel -:C -oxyz --gen3d",
]

[tool.cibuildwheel.linux]
before-all = [
    "yum install -y zlib-devel-1.2.7 eigen3-devel-3.3.7 libxml2-devel-2.9.1",
]

and I use Github actions to build everything.

The issue I have is that the wheel builds fine:

truncated install logs 
2024-08-23T08:19:42.0620747Z   *** Installing project into wheel...
2024-08-23T08:19:42.1426718Z   -- Install configuration: "Release"
2024-08-23T08:19:42.1427901Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel
2024-08-23T08:19:42.1437400Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel/pybel.py
...
2024-08-23T08:19:42.2599648Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0
2024-08-23T08:19:42.2607617Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7
2024-08-23T08:19:42.2614782Z   -- Set non-toolchain portion of runtime path of "/tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0" to "/tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/lib"
2024-08-23T08:19:42.2621652Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so
...
2024-08-23T08:19:42.4828912Z   *** Making wheel...
2024-08-23T08:19:45.5498324Z   *** Created aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:19:45.5688505Z   Building wheel for aqemia-openbabel (pyproject.toml): finished with status 'done'
2024-08-23T08:19:45.5786590Z   Created wheel for aqemia-openbabel: filename=aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl size=11102455 sha256=363d1d05dad22183022cd9e4f7c9f8a62d912bc44394f8835c03ef171b461551
2024-08-23T08:19:45.5788988Z   Stored in directory: /root/.cache/pip/wheels/21/14/79/baf708a1463c156ec37e5504ed91f66de9ca8b1bc844f71781
2024-08-23T08:19:45.5824955Z Successfully built aqemia-openbabel
2024-08-23T08:19:45.5940993Z Removed build tracker: '/tmp/pip-build-tracker-q2u7ftms'
2024-08-23T08:19:45.6790642Z     + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
2024-08-23T08:19:45.7065003Z     + rm -rf /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7091788Z     + mkdir -p /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7124575Z ##[endgroup]

however the auditwheel repair fails as it cannot find libraries since they only live in the wheel file

logs 
2024-08-23T08:19:45.7126351Z ##[group]Repairing wheel...
2024-08-23T08:19:45.7126696Z 
2024-08-23T08:19:45.7130107Z     + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'
2024-08-23T08:19:46.0501385Z INFO:auditwheel.main_repair:Repairing aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:20:06.1838904Z Traceback (most recent call last):
2024-08-23T08:20:06.1870673Z   File "/usr/local/bin/auditwheel", line 8, in <module>
2024-08-23T08:20:06.1871685Z     sys.exit(main())
2024-08-23T08:20:06.1873139Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main.py", line 54, in main
2024-08-23T08:20:06.1874430Z     rval = args.func(args, p)
2024-08-23T08:20:06.1875837Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main_repair.py", line 173, in execute
2024-08-23T08:20:06.1877130Z     out_wheel = repair_wheel(
2024-08-23T08:20:06.1878534Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/repair.py", line 78, in repair_wheel
2024-08-23T08:20:06.1879819Z     raise ValueError(
2024-08-23T08:20:06.1880749Z ValueError: Cannot repair wheel, because required library "libopenbabel.so.7" could not be located
2024-08-23T08:20:06.5140592Z ##[endgroup]
2024-08-23T08:20:06.5147019Z                                                              ✕ 20.80s
2024-08-23T08:20:06.5184462Z ##[error]Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'] failed with code 1.

I believe I must be missing something obvious here but I don't see how to fix the problem except by using a before-all action that basically builds the project. This doubles the build time which is not great. Would there be other options that I missed?

Wheel configuration through pyproject

Also I'm struggling defining the wheels I want to build. I'm only interested in building cp311-manylinux2014-x86_64.
When I set my pyproject.toml (which is what I understand as the preferred way) to

 [tool.cibuildwheel]
 archs = ["x86_64"]
 build = ["cp311-manylinux2014-x86_64"]
 platform = ["linux"]

it seems that cibuildwheel does't pick the configuration and the action run fails with

cibuildwheel: No build identifiers selected: BuildSelector(build_config='cp311-manylinux2014-x86_64', skip_config='', requires_python=<SpecifierSet('<3.12,>=3.11')>, prerelease_pythons=False, free_threaded_support=False)

So I end up defining CIBW_ARCHS, CIWB_BUILD and CIWB_PLATFORM which I would rather not. Is there some precedence rule that I misunderstood?

Thanks a lot for any help!

Build log

No response

CI config

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the named CMakeLists.txt and pyproject.toml, then reproduce the cibuildwheel run from the auditwheel repair log. Trace how libraries installed into the wheel are discovered during repair and how the pyproject build, arch, and platform settings are selected. Done means the reported wheel configuration is honored and the built wheel reaches repair without the missing libopenbabel.so.7 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, github-actions, python
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.