conda-forge / conda-forge/arrow-cpp-feedstock

Importing pyarrow on Windows fails when calling from Julia via PythonCall

Open
#1,176 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
12
Forks
66
PR merge metrics
No merged PRs in 30d

Description

### Solution to issue cannot be found in the documentation.

- [X] I checked the documentation.

### Issue

When I try to import pyarrow via PythonCall from Julia, I receive an error if pyarrow is installed via conda.
MWE:
Create a clean environment:
```
conda create -n pyarrow_conda -c conda-forge pyarrow
```
and then execute the following Julia code
```julia
julia> ENV["JULIA_PYTHONCALL_EXE"] = "C:\\ProgramData\\mambaforge\\envs\\pyarrow_conda\\python.exe"
"C:\\ProgramData\\mambaforge\\envs\\pyarrow_conda\\python.exe"

julia> using PythonCall

julia> pyimport("pyarrow")
ERROR: Python: ImportError: DLL load failed while importing lib: Die angegebene Prozedur wurde nicht gefunden.
Python stacktrace:
[1]
@ C:\ProgramData\mambaforge\envs\pyarrow_conda\Lib\site-packages\pyarrow\__init__.py:65
Stacktrace:
[1] pythrow()
@ PythonCall C:\Users\\.julia\dev\PythonCall\src\err.jl:94
[2] errcheck
@ C:\Users\\.julia\dev\PythonCall\src\err.jl:10 [inlined]
[3] pyimport(m::String)
@ PythonCall C:\Users\\.julia\dev\PythonCall\src\concrete\import.jl:11
[4] top-level scope
@ REPL[3]:1
```
However, if I install pyarrow via pip, no error is produced.
```julia
julia> ENV["JULIA_PYTHONCALL_EXE"] = "C:\\ProgramData\\mambaforge\\envs\\pyarrow_pip\\python.exe"
"C:\\ProgramData\\mambaforge\\envs\\pyarrow_pip\\python.exe"

julia> using PythonCall

julia> pyimport("pyarrow")
Python:
```
On a ubuntu linux, no error occurs for conda installations of pyarrow.

The file __init.py is identical in both conda and pip installation.

Any hint where this could come from?

### Installed packages

```shell
Difference between pyarrow_pip and pyarrow_conda:

pyarrow 13.0.0 py311h6a6099b_4_cpu conda-forge
pyarrow 13.0.0 pypi_0 pypi

#-----------------------------------------------------------------------

Detailed listings:

(pyarrow_conda) PS C:\Users\> conda list
# packages in environment at C:\ProgramData\mambaforge\envs\pyarrow_conda:
#
# Name Version Build Channel
aws-c-auth 0.7.3 hd125877_3 conda-forge
aws-c-cal 0.6.2 h04c9df6_1 conda-forge
aws-c-common 0.9.0 hcfcfb64_0 conda-forge
aws-c-compression 0.2.17 h04c9df6_2 conda-forge
aws-c-event-stream 0.3.2 h495bb32_0 conda-forge
aws-c-http 0.7.12 h0890e15_1 conda-forge
aws-c-io 0.13.32 h83b3346_3 conda-forge
aws-c-mqtt 0.9.6 h0fd1aac_0 conda-forge
aws-c-s3 0.3.17 h9f49523_0 conda-forge
aws-c-sdkutils 0.1.12 h04c9df6_1 conda-forge
aws-checksums 0.1.17 h04c9df6_1 conda-forge
aws-crt-cpp 0.23.1 hfd73fcf_2 conda-forge
aws-sdk-cpp 1.11.156 h03febf0_2 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
c-ares 1.19.1 hcfcfb64_0 conda-forge
ca-certificates 2023.7.22 h56e8100_0 conda-forge
intel-openmp 2023.2.0 h57928b3_49503 conda-forge
krb5 1.21.2 heb0366b_0 conda-forge
libabseil 20230802.1 cxx17_h63175ca_0 conda-forge
libarrow 13.0.0 h1e3473c_4_cpu conda-forge
libblas 3.9.0 18_win64_mkl conda-forge
libbrotlicommon 1.1.0 hcfcfb64_0 conda-forge
libbrotlidec 1.1.0 hcfcfb64_0 conda-forge
libbrotlienc 1.1.0 hcfcfb64_0 conda-forge
libcblas 3.9.0 18_win64_mkl conda-forge
libcrc32c 1.1.2 h0e60522_0 conda-forge
libcurl 8.3.0 hd5e4a3a_0 conda-forge
libevent 2.1.12 h3671451_1 conda-forge
libexpat 2.5.0 h63175ca_1 conda-forge
libffi 3.4.2 h8ffe710_5 conda-forge
libgoogle-cloud 2.12.0 h0a0a397_2 conda-forge
libgrpc 1.57.0 h550f6bd_1 conda-forge
libhwloc 2.9.3 default_haede6df_1009 conda-forge
libiconv 1.17 h8ffe710_0 conda-forge
liblapack 3.9.0 18_win64_mkl conda-forge
libprotobuf 4.23.4 hb8276f3_6 conda-forge
libsqlite 3.43.0 hcfcfb64_0 conda-forge
libssh2 1.11.0 h7dfc565_0 conda-forge
libthrift 0.19.0 h06f6336_0 conda-forge
libutf8proc 2.8.0 h82a8f57_0 conda-forge
libxml2 2.11.5 hc3477c8_1 conda-forge
libzlib 1.2.13 hcfcfb64_5 conda-forge
lz4-c 1.9.4 hcfcfb64_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
numpy 1.26.0 py311h0b4df5a_0 conda-forge
openssl 3.1.3 hcfcfb64_0 conda-forge
orc 1.9.0 h8dbeef6_2 conda-forge
pip 23.2.1 pyhd8ed1ab_0 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
pyarrow 13.0.0 py311h6a6099b_4_cpu conda-forge
python 3.11.5 h2628c8c_0_cpython conda-forge
python_abi 3.11 4_cp311 conda-forge
re2 2023.03.02 hd4eee63_0 conda-forge
setuptools 68.2.2 pyhd8ed1ab_0 conda-forge
snappy 1.1.10 hfb803bf_0 conda-forge
tbb 2021.10.0 h91493d7_1 conda-forge
tk 8.6.13 hcfcfb64_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ucrt 10.0.22621.0 h57928b3_0 conda-forge
vc 14.3 h64f974e_17 conda-forge
vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge
vs2015_runtime 14.36.32532 h05e6639_17 conda-forge
wheel 0.41.2 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
zstd 1.5.5 h12be248_0 conda-forge

# -------------------------------------------------------------------------

(pyarrow_pip) PS C:\Users\> conda list
# packages in environment at C:\ProgramData\mambaforge\envs\pyarrow_pip:
#
# Name Version Build Channel
aws-c-auth 0.7.3 hd125877_3 conda-forge
aws-c-cal 0.6.2 h04c9df6_1 conda-forge
aws-c-common 0.9.0 hcfcfb64_0 conda-forge
aws-c-compression 0.2.17 h04c9df6_2 conda-forge
aws-c-event-stream 0.3.2 h495bb32_0 conda-forge
aws-c-http 0.7.12 h0890e15_1 conda-forge
aws-c-io 0.13.32 h83b3346_3 conda-forge
aws-c-mqtt 0.9.6 h0fd1aac_0 conda-forge
aws-c-s3 0.3.17 h9f49523_0 conda-forge
aws-c-sdkutils 0.1.12 h04c9df6_1 conda-forge
aws-checksums 0.1.17 h04c9df6_1 conda-forge
aws-crt-cpp 0.23.1 hfd73fcf_2 conda-forge
aws-sdk-cpp 1.11.156 h03febf0_2 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
c-ares 1.19.1 hcfcfb64_0 conda-forge
ca-certificates 2023.7.22 h56e8100_0 conda-forge
intel-openmp 2023.2.0 h57928b3_49503 conda-forge
krb5 1.21.2 heb0366b_0 conda-forge
libabseil 20230802.1 cxx17_h63175ca_0 conda-forge
libarrow 13.0.0 h1e3473c_4_cpu conda-forge
libblas 3.9.0 18_win64_mkl conda-forge
libbrotlicommon 1.1.0 hcfcfb64_0 conda-forge
libbrotlidec 1.1.0 hcfcfb64_0 conda-forge
libbrotlienc 1.1.0 hcfcfb64_0 conda-forge
libcblas 3.9.0 18_win64_mkl conda-forge
libcrc32c 1.1.2 h0e60522_0 conda-forge
libcurl 8.3.0 hd5e4a3a_0 conda-forge
libevent 2.1.12 h3671451_1 conda-forge
libexpat 2.5.0 h63175ca_1 conda-forge
libffi 3.4.2 h8ffe710_5 conda-forge
libgoogle-cloud 2.12.0 h0a0a397_2 conda-forge
libgrpc 1.57.0 h550f6bd_1 conda-forge
libhwloc 2.9.3 default_haede6df_1009 conda-forge
libiconv 1.17 h8ffe710_0 conda-forge
liblapack 3.9.0 18_win64_mkl conda-forge
libprotobuf 4.23.4 hb8276f3_6 conda-forge
libsqlite 3.43.0 hcfcfb64_0 conda-forge
libssh2 1.11.0 h7dfc565_0 conda-forge
libthrift 0.19.0 h06f6336_0 conda-forge
libutf8proc 2.8.0 h82a8f57_0 conda-forge
libxml2 2.11.5 hc3477c8_1 conda-forge
libzlib 1.2.13 hcfcfb64_5 conda-forge
lz4-c 1.9.4 hcfcfb64_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
numpy 1.26.0 py311h0b4df5a_0 conda-forge
openssl 3.1.3 hcfcfb64_0 conda-forge
orc 1.9.0 h8dbeef6_2 conda-forge
pip 23.2.1 pyhd8ed1ab_0 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
pyarrow 13.0.0 pypi_0 pypi
python 3.11.5 h2628c8c_0_cpython conda-forge
python_abi 3.11 4_cp311 conda-forge
re2 2023.03.02 hd4eee63_0 conda-forge
setuptools 68.2.2 pyhd8ed1ab_0 conda-forge
snappy 1.1.10 hfb803bf_0 conda-forge
tbb 2021.10.0 h91493d7_1 conda-forge
tk 8.6.13 hcfcfb64_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ucrt 10.0.22621.0 h57928b3_0 conda-forge
vc 14.3 h64f974e_17 conda-forge
vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge
vs2015_runtime 14.36.32532 h05e6639_17 conda-forge
wheel 0.41.2 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
zstd 1.5.5 h12be248_0 conda-forge
```

### Environment info

```shell
active environment : pyarrow_conda
active env location : C:\ProgramData\mambaforge\envs\pyarrow_conda
shell level : 4
user config file : C:\Users\\.condarc
populated config files : C:\ProgramData\mambaforge\.condarc
C:\Users\\.condarc
conda version : 23.3.1
conda-build version : not installed
python version : 3.10.12.final.0
virtual packages : __archspec=1=x86_64
__cuda=12.1=0
__win=0=0
base environment : C:\ProgramData\mambaforge (writable)
conda av data dir : C:\ProgramData\mambaforge\etc\conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/win-64
https://conda.anaconda.org/conda-forge/noarch
package cache : C:\ProgramData\mambaforge\pkgs
envs directories : C:\ProgramData\mambaforge\envs
platform : win-64
user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Windows/10 Windows/10.0.19045
administrator : False
netrc file : None
offline mode : False
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.