[R][Python] Extension types cannot be registered in both R and Python
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
When registering extension types as is now possible in the R bindings, it looks as though we cannot register an extension type in R and Python at the same time:
```R
# apache/arrow@master
library(arrow, warn.conflicts = FALSE)
library(reticulate)
# this is a virtualenv with pyarrow installed against the same commit
use_virtualenv(
"/Users/deweydunnington/Desktop/rscratch/pyarrow-dev",
required = TRUE
)
pa <- import("pyarrow")
pa[["__version__"]]
#> [1] "9.0.0.dev131+g8a36f0f6c"
py_run_string("
import pyarrow as pa
class TestExtensionType(pa.ExtensionType):
def __init__(self):
super().__init__(pa.int32(), 'arrow.test_type')
def __arrow_ext_serialize__(self):
return b''
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
return cls()
pa.register_extension_type(TestExtensionType())
")
arrow::register_extension_type(
arrow::new_extension_type(int32(), "arrow.test_type")
)
#> Error: Key error: A type extension with name arrow.test_type already defined
```
I also get a segfault if I try to surface a Python type into R (probably because the R bindings mistakenly assume that if `type.id() == Type::EXTENSION` then it is safe to cast to our own `ExtensionType` C++ subclass that implements R-specific things.
This came about because the 'geoarrow' Python and 'geoarrow' R packages both register a number of extension type definitions.
- geoarrow's Python registration: https://github.com/jorisvandenbossche/python-geoarrow/blob/main/src/geoarrow/extension_types.py#L108-L117
- geoarrow's R registration: https://github.com/paleolimbot/geoarrow/blob/master/R/pkg-arrow.R#L208-L223
I can also force an interaction if I build GDAL against the same Arrow that the arrow R package is linked against and attempt to load a Feather file saved with an extension type using the sf package. I will attempt to recreate that interaction as well in both R and Python.
I don't know enough about linking to know to what extent this is linked to my own development setup/build of the R package, although I think there are at least some environments where a shared library is picked up first by the R config script (fedora36, for example). It does look like my own R package build is dynamically linking to libarrow.dylib.
**Reporter**: [Dewey Dunnington](https://issues.apache.org/jira/browse/ARROW-16688) / @paleolimbot
**Note**: *This issue was originally created as [ARROW-16688](https://issues.apache.org/jira/browse/ARROW-16688). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the R and Python registration examples in the issue, then inspect the linked geoarrow registration entry points in Python and R. Compare how extension types are registered and surfaced across both bindings, including the reported cast and shared-library scenarios. Done means compatible registration works without the key error or segfault.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, r
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100