[Python] pyarrow 21.0.0 causes "NumPy requires GCC >= 9.3" error, even if I pin numpy<2.3.0
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I am using pyarrow in a Python 3.11 AWS Lambda function.
This Python version means the OS is Amazon Linux 2, which in turn implies a particular GCC version.
I started getting failures in my CI pipeline when we allowed pyarrow to update to latest 21.0.0 version:
```
19.18 ✔ Install numpy 2.2.6 successful
...
38.85 ✖ Install pyarrow 21.0.0 failed
38.86
38.86 ERRORS:
38.86 add pyarrow failed:
38.86 Traceback (most recent call last):
38.86 File "/var/lang/lib/python3.11/concurrent/futures/thread.py", line 58, in run
38.86 result = self.fn(*self.args, **self.kwargs)
38.86 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/installers/synchronizers.py",
38.86 line 29, in install_candidate
38.86 self.manager.install(can)
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/installers/manager.py", line
38.86 33, in install
38.86 prepared.build(),
38.86 ^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/models/candidates.py", line
38.86 426, in build
38.86 self._cached = Path(builder.build(build_dir,
38.86 metadata_directory=self._metadata_dir))
38.86 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.86 ^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/base.py", line 84,
38.86 in wrapper
38.86 return func(*args, **kwargs)
38.86 ^^^^^^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/wheel.py", line 23,
38.86 in build
38.86 self.install(self._requires, shared=****)
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/base.py", line 323,
38.86 in install
38.86 install_requirements(missing, env, allow_uv=*****)
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/installers/core.py", line 45,
38.86 in install_requirements
38.86 syncer.synchronize()
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/installers/base.py", line
38.86 221, in synchronize
38.86 manager.install(can)
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/installers/manager.py", line
38.86 33, in install
38.86 prepared.build(),
38.86 ^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/models/candidates.py", line
38.86 426, in build
38.86 self._cached = Path(builder.build(build_dir,
38.86 metadata_directory=self._metadata_dir))
38.86 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.86 ^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/base.py", line 84,
38.86 in wrapper
38.86 return func(*args, **kwargs)
38.86 ^^^^^^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/wheel.py", line 26,
38.86 in build
38.86 filename = self._hook.build_wheel(out_dir, self.config_settings,
38.86 metadata_directory)
38.86 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.86 ^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line
38.86 256, in build_wheel
38.86 return self._call_hook(
38.86 ^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pyproject_hooks/_impl.py", line
38.86 392, in _call_hook
38.86 self._subprocess_runner(
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/base.py", line 281,
38.86 in subprocess_runner
38.86 return log_subprocessor(cmd, cwd, extra_environ=env)
38.86 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.86 File "/var/lang/lib/python3.11/site-packages/pdm/builders/base.py", line 128,
38.86 in log_subprocessor
38.86 raise build_error(e) from None
38.86 pdm.exceptions.BuildError: Build backend raised error: Showing the last 10 lines
38.86 of the build output:
38.86 C linker for the host machine: cc ld.bfd 2.29.1-31
38.86 C++ compiler for the host machine: c++ (gcc 7.3.1 "c++ (GCC) 7.3.1 20180712 (Red
38.86 Hat 7.3.1-17)")
38.86 C++ linker for the host machine: c++ ld.bfd 2.29.1-31
38.86 Cython compiler for the host machine: cython (cython 3.1.4)
38.86 Host machine cpu family: aarch64
38.86 Host machine cpu: aarch64
38.86
38.86 ../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 9.3
```
AFAIK this issue is associated with numpy `2.3.0` which starts to depend on GCC >= 9.3
We are using [pdm](https://pdm-project.org/) as our installer. In our `pyproject.toml` we already pin `"numpy<2.3.0"`
The build output reports:
```
19.18 ✔ Install numpy 2.2.6 successful
...
38.85 ✖ Install pyarrow 21.0.0 failed
```
However if we also pin `"pyarrow<21.0.0"` then the build succeeds.
I didn't see anything obvious in the [release notes](https://arrow.apache.org/release/21.0.0.html) so I am hoping this can be avoided.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.