apache / apache/arrow

How to expose missing C++ API to Cython?

Open
#14,421 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hello,
not sure if this is 100% a question for Arrow, but I wanted to experiment a bit with Cython, `nogil` and `arrow`. So I started writing a small function that uses `GetColumnByName`, that is not exposed for what I can see in `https://github.com/apache/arrow/blob/master/python/pyarrow/includes/libarrow.pxd`.

```
import pyarrow as pa

from libcpp.string cimport string
from libcpp.memory cimport shared_ptr

from pyarrow.lib cimport CTable, pyarrow_unwrap_table(sa_table)

cdef string SCORES_COLUMN = b"SCORES_COLUMNS"

cdef extern from "arrow/api.h" namespace "arrow" nogil:
cdef cppclass CTable" arrow::Table":
shared_ptr[CChunkedArray] GetColumnByName(const string&)

def normalize_sa(sa_table: pa.Table) -> None:
unwrapped_table: shared_ptr[CTable] = pyarrow_unwrap_table(sa_table)

unwrapped_table.get().GetColumnByName(SCORES_COLUMN)

```

But when compiling this, I get:
```
Error compiling Cython file:
------------------------------------------------------------
...
def normalize_sa(sa_table: pa.Table) -> None:
unwrapped_table: shared_ptr[CTable] = pyarrow_unwrap_table(sa_table)

unwrapped_table.get().GetColumnByName(SCORES_COLUMN)
^
------------------------------------------------------------

src\pkg\cutils\_normalize.pyx:35:25: Object of type 'CTable' has no attribute 'GetColumnByName'
```

Could you please help me understand why I can't expose `GetColumnByName` or why cython can't find it?

Contributor guide

Open the contributing guide

Research direction

Compare the CTable declaration in python/pyarrow/includes/libarrow.pxd with GetColumnByName in arrow/api.h, then reproduce the reported Cython compile error from src/pkg/cutils/_normalize.pyx. The work is complete when the method can be resolved from the Cython binding, or the binding limitation is documented with a clear explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.