apache / apache/arrow

[R][Python] Unable to load pyarrow package through reticulate in R on windows

Open
#40,073 8 comments 3 reactions 1 assignee Claimed by @amoeba View on GitHub
Component: Python Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to use arrow to share large data frames between python and R. If I load the R packages then try to import the python module this fails with an ImportError from python:

```
> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

timestamp

> library(reticulate)
> py_discover_config()
...
version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:29:04) [MSC v.1929 64 bit (AMD64)]
Architecture: 64bit
...
> pa <- import("pyarrow")
Error in py_module_import(module, convert = convert) :
ImportError: DLL load failed while importing lib: The specified procedure could not be found.
Run `reticulate::py_last_error()` for details.

```

Note that I can load the pyarrow in a vanilla python REPL just fine.

If I ensure to import pyarrow before loading the R package then it works, however this is difficult to guarantee (other packages may have already loaded the R arrow package), and causes failures during package development.

```
> library(reticulate)
> import('pyarrow')
Module(pyarrow)
> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

timestamp

```

Because it seems to be driven by some sort of incompatibility between the R and python arrow libraries, I tried various different versions to see if I could get anything that works. Initially I was using R 4.2.1, but could also reproduce with a fresh install of R 4.3.2. I tried various python versions including fresh installs of the latest (3.11.5), and also the conda version as installed by `reticulate::install_miniconda` (3.9.18 above). I tried using the latest version of the pyarrow package (15.0.0), and also downgrading to 14.0.0 to match the latest available R package.

### Component(s)

Python, R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.