pypa / pypa/setuptools

[BUG] breaks numpy in cross-compiling environment

Open
#3,549 10 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Needs Triage
Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

setuptools version

setuptools >= 64.x

Python version

python 3.10.x

OS

Linux

Additional environment information

Environment is using the Synology DSM toolchain for cross-compiling for multiple archs (armv5, v7, v8, ppc, x64, i686).

Description

While preparing python 3.10.6 update I've took care to update all build requirements, including setuptools.
It hapens that it made numpy to fail to build starting with setuptools >= 64.x (also fails on 65.x).
Full build logs can be found here: https://github.com/SynoCommunity/spksrc/runs/7917533394?check_suite_focus=true
Associated SynoCommunity PR: https://github.com/SynoCommunity/spksrc/pull/5395

Expected behavior

numpy to build properly (like all other regular wheel being cross-compiled)

How to Reproduce

Can be reproduced easily using the SynoCommunity (spksrc) cross-compiling environment.

Output

Output is mostly hidden under pip call:

===>  _PYTHON_HOST_PLATFORM=x86_64-pc-linux-gnu /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/cross/bin/pip wheel --disable-pip-version-check --no-binary :all: --cache-dir /home/spksrc/py310.6/spksrc/spk/bazarr/../../distrib/pip --no-deps --wheel-dir /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/wheelhouse --no-build-isolation greenlet==1.1.2
Collecting greenlet==1.1.2
  Using cached greenlet-1.1.2.tar.gz (91 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: greenlet
  Building wheel for greenlet (setup.py): started
  Building wheel for greenlet (setup.py): finished with status 'done'
  Created wheel for greenlet: filename=greenlet-1.1.2-cp310-cp310-linux_x86_64.whl size=142558 sha256=d3ac376cd9b322f76ee60208a103be9f52d989d3b3a8a1185fb9857c4a28de25
  Stored in directory: /home/spksrc/py310.6/spksrc/distrib/pip/wheels/51/01/b7/16995feb30f4eed2504aed2260a4058a1164472c9be5ab44b1
Successfully built greenlet
make[3]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
===>  [numpy]           
make[3]: Entering directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
===>  _PYTHON_HOST_PLATFORM=x86_64-pc-linux-gnu /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/cross/bin/pip wheel --disable-pip-version-check --no-binary :all: --cache-dir /home/spksrc/py310.6/spksrc/spk/bazarr/../../distrib/pip --no-deps --wheel-dir /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/wheelhouse --no-build-isolation numpy==1.22.4
Collecting numpy==1.22.4
  Using cached numpy-1.22.4.zip (11.5 MB)
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [75 lines of output]
      Running from numpy source directory.
      running dist_info
      running build_src
      INFO: build_src
      creating /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info
      writing /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/entry_points.txt
      writing top-level names to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/SOURCES.txt'
      /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py:643: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
      Please extend command classes from setuptools instead of distutils.
        warnings.warn(
      INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
      INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
      Traceback (most recent call last):
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 369, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 474, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 334, in run_setup
          exec(code, locals())
        File "<string>", line 461, in <module>
        File "<string>", line 453, in setup_package
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
          self.run_command(cmd)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/dist_info.py", line 96, in run
          self.egg_info.run()
        File "/tmp/pip-wheel-zx645048/numpy_62d5a1a8355144f781d4c843ede91f36/numpy/distutils/command/egg_info.py", line 25, in run
          _egg_info.run(self)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 308, in run
          self.find_sources()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 316, in find_sources
          mm.run()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 560, in run
          self.add_defaults()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 597, in add_defaults
          sdist.add_defaults(self)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 107, in add_defaults
          self._add_defaults_build_sub_commands()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 127, in _add_defaults_build_sub_commands
          self.filelist.extend(chain.from_iterable(files))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 503, in extend
          self.files.extend(filter(self._safe_path, paths))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 126, in <genexpr>
          files = (c.get_source_files() for c in cmds if hasattr(c, "get_source_files"))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 125, in <genexpr>
          cmds = (self.get_finalized_command(c) for c in missing_cmds)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 306, in get_finalized_command
          cmd_obj.ensure_finalized()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 109, in ensure_finalized
          self.finalize_options()
        File "/tmp/pip-wheel-zx645048/numpy_62d5a1a8355144f781d4c843ede91f36/numpy/distutils/command/config_compiler.py", line 69, in finalize_options
          v = getattr(c, a)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 157, in __getattr__
          return _build_ext.build_ext.__getattr__(self, name)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 105, in __getattr__
          raise AttributeError(attr)
      AttributeError: fcompiler. Did you mean: 'compiler'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
make[3]: *** [../../mk/spksrc.wheel.mk:139: cross-compile-wheel-numpy] Error 1
make[3]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make[2]: *** [../../mk/spksrc.wheel.mk:81: build_wheel_target] Error 1
make[2]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make[1]: *** [../../mk/spksrc.spk.mk:603: build-arch-x64-6.1] Error 1
make[1]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make: *** [../../mk/spksrc.spk.mk:597: arch-x64-6.1] Error 2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cross-compiling pip wheel command and the traceback ending in numpy/distutils/command/config_compiler.py and setuptools/command/sdist.py. Reproduce the numpy==1.22.4 build with setuptools 64.x or newer in the SynoCommunity environment; done means numpy builds successfully across the listed target architectures.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.