apache / apache/arrow-nanoarrow

DISC: Prefer nanobind to Cython

Open
#597 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

Wanted to open this as a topic for discussion! I've used nanobind a lot lately, and find it to be a very good library. Cython of course, has been widely used in scientific Python projects for some time, but I think suffers from a few usability issues:

1. The Cython debugger is broken, and has been for a long time
2. IDE support is limited
3. The syntax for mixing in raw C code is rather awkward (ex: getting the pointer to the start of an array in Cython)
4. Performance benefits are not always clear, and subtle differences in declarations can have drastic impacts (Cython annotations can help you inspect this, if you use them)
5. Specific to nanoarrow, the build system has a rather complicated setup whereby it generates Cython header files from nanoarrow sources

Nanobind rather natively solves issues 1-3 above. Point 4 is partially solved by nanobind; a missing declaration wouldn't be the culprit for poor performance, but you still would need to understand the impacts of interaction with Python objects. For point 5, nanobind was also recently added to the Meson wrapdb, so if nanoarrow decided to use meson + meson-python it could drastically simplify the process of building Python extensions.

Of course, the downside to nanobind is that you are trading Python syntax for C++, which may be off-putting to some developers. Taking `CArrayBuilder.append_bytes` as an example, the current code looks like (N.B. the current code probably should declare `code` to be of type `ArrowErrorCode` but does not):

```python
def append_bytes(self, obj: Iterable[Union[str, None]]) -> CArrayBuilder:
cdef Py_buffer buffer
cdef ArrowBufferView item

for py_item in obj:
if py_item is None:
code = ArrowArrayAppendNull(self._ptr, 1)
else:
PyObject_GetBuffer(py_item, &buffer, PyBUF_ANY_CONTIGUOUS | PyBUF_FORMAT)

if buffer.ndim != 1:
raise ValueError("Can't append buffer with dimensions != 1 to binary array")

if buffer.itemsize != 1:
PyBuffer_Release(&buffer)
raise ValueError("Can't append buffer with itemsize != 1 to binary array")

item.data.data = buffer.buf
item.size_bytes = buffer.len
code = ArrowArrayAppendBytes(self._ptr, item)
PyBuffer_Release(&buffer)

if code != NANOARROW_OK:
Error.raise_error(f"append bytes item {py_item}", code)
```

in nanobind, an equivalent implementation probably looks like this (N.B. this is untested):

```c++
auto AppendBytes(const CArrayBuilder &builder, nb::iterable obj)
ArrowErrorCode code;
ArrowBufferView item

for (const auto &py_item : obj) {
if (py_item.is_none()) {
code = ArrowArrayAppendNull(ptr_, 1);
} else {
Py_buffer buffer;
PyObject_GetBuffer(py_item, &buffer, PyBUF_ANY_CONTIGUOUS | PyBUF_FORMAT);

if (buffer.ndim != 1)
throw nb::value_error("Can't append buffer with dimensions != 1 to binary array");

if (buffer.itemsize != 1) {
PyBuffer_Release(&buffer);
throw nb::value_error("Can't append buffer with itemsize != 1 to binary array");
}

item.data.data = buffer.buf;
item.size_bytes = buffer.len;
code = ArrowArrayAppendBytes(ptr_, item);
PyBuffer_Release(&buffer);
}

if (code != NANOARROW_OK)
// TODO: would need to define NanoarrowException class as subclass of RuntimeError
throw NanoarrowException("append bytes item " + py_item.str() + " failed " + std::to_string(code));
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

This is a discussion rather than an implementation task, and it names no files or tests. Start by reviewing the current CArrayBuilder.append_bytes implementation and the Cython-generated nanoarrow headers/build setup; a concrete migration scope and acceptance criteria would be needed before defining what done means.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
build-system, developer-experience
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.