apache / apache/arrow

[Python] `test_cpp_extension_in_python` started failing on osx in conda-forge

Open
#43,800 2 comments 0 reactions 0 assignees View on GitHub
Component: Packaging Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

This is probably related to some ambient change, because it happened across the board from 17.x down to 14.x.

Failure looks like

```
=================================== FAILURES ===================================
_________________________ test_cpp_extension_in_python _________________________

tmpdir = local('/private/var/folders/rw/0gwtqtr55hdbnkwp7bt6pkb00000gn/T/pytest-of-runner/pytest-0/test_cpp_extension_in_python0')

@pytest.mark.cython
def test_cpp_extension_in_python(tmpdir):
from .test_cython import (
setup_template, compiler_opts, test_ld_path, test_util, here)
with tmpdir.as_cwd():
# Set up temporary workspace
pyx_file = 'extensions.pyx'
shutil.copyfile(os.path.join(here, pyx_file),
os.path.join(str(tmpdir), pyx_file))
# Create setup.py file
setup_code = setup_template.format(pyx_file=pyx_file,
compiler_opts=compiler_opts,
test_ld_path=test_ld_path)
with open('setup.py', 'w') as f:
f.write(setup_code)

subprocess_env = test_util.get_modified_env_with_pythonpath()

# Compile extension module
> subprocess.check_call([sys.executable, 'setup.py',
'build_ext', '--inplace'],
env=subprocess_env)

pyarrow/tests/test_extension_type.py:1342:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['$PREFIX/bin/python3.12', 'setup.py', 'build_ext', '--inplace'],)
kwargs = {'env': {'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/Users/runner/actionarchivecache', 'AGENT_ACCEPTTEEEULA': 'True', 'AGENT_BUILDDIRECTORY': '/Users/runner/work/1', 'AGENT_CLOUDID': '62d17dae-1107-4d91-a05c-4105b32adb0e', ...}}
retcode = 1
cmd = ['$PREFIX/bin/python3.12', 'setup.py', 'build_ext', '--inplace']

def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.

The arguments are the same as for the call function. Example:

check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E subprocess.CalledProcessError: Command '['$PREFIX/bin/python3.12', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1.

../subprocess.py:413: CalledProcessError
----------------------------- Captured stdout call -----------------------------
Compiling extensions.pyx because it changed.
[1/1] Cythonizing extensions.pyx
Extension module: ['$PREFIX/lib/python3.12/site-packages/numpy/_core/include', '$PREFIX/lib/python3.12/site-packages/pyarrow/include'] ['arrow_python', 'arrow'] ['$PREFIX/lib/python3.12/site-packages/pyarrow']
running build_ext
building 'extensions' extension
creating build
creating build/temp.macosx-10.13-x86_64-cpython-312
clang++ -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -Wall -fPIC -O2 -isystem $PREFIX/include -fPIC -O2 -isystem $PREFIX/include -I$PREFIX/lib/python3.12/site-packages/numpy/_core/include -I$PREFIX/lib/python3.12/site-packages/pyarrow/include -I$PREFIX/include/python3.12 -c extensions.cpp -o build/temp.macosx-10.13-x86_64-cpython-312/extensions.o -std=c++17
creating build/lib.macosx-10.13-x86_64-cpython-312
clang++ -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -Wall -fPIC -O2 -isystem $PREFIX/include -fPIC -O2 -isystem $PREFIX/include -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-cpython-312/extensions.o -L$PREFIX/lib/python3.12/site-packages/pyarrow -larrow_python -larrow -o build/lib.macosx-10.13-x86_64-cpython-312/extensions.cpython-312-darwin.so
----------------------------- Captured stderr call -----------------------------
[removed compilation warnings]

27 warnings generated.
ld: library not found for -larrow
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang++' failed with exit code 1
```

Looking at the link path, it looks like there's only `-L$PREFIX/lib/python3.12/site-packages/pyarrow` being passed, whereas `libarrow.{so,dylib}` is actually in `$PREFIX/lib`. It's IMO possible/likely that this is a change or regression in setuptools (the aboved used 72.2.0).

### Component(s)

Packaging, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.