apache / apache/arrow-nanoarrow
DISC: Prefer nanobind to Cython
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 11
Description
Wanted to open this as a topic for discussion! I've used nanobind a lot lately, and find it to be a very good library. Cython of course, has been widely used in scientific Python projects for some time, but I think suffers from a few usability issues:
1. The Cython debugger is broken, and has been for a long time
2. IDE support is limited
3. The syntax for mixing in raw C code is rather awkward (ex: getting the pointer to the start of an array in Cython)
4. Performance benefits are not always clear, and subtle differences in declarations can have drastic impacts (Cython annotations can help you inspect this, if you use them)
5. Specific to nanoarrow, the build system has a rather complicated setup whereby it generates Cython header files from nanoarrow sources
Nanobind rather natively solves issues 1-3 above. Point 4 is partially solved by nanobind; a missing declaration wouldn't be the culprit for poor performance, but you still would need to understand the impacts of interaction with Python objects. For point 5, nanobind was also recently added to the Meson wrapdb, so if nanoarrow decided to use meson + meson-python it could drastically simplify the process of building Python extensions.
Of course, the downside to nanobind is that you are trading Python syntax for C++, which may be off-putting to some developers. Taking `CArrayBuilder.append_bytes` as an example, the current code looks like (N.B. the current code probably should declare `code` to be of type `ArrowErrorCode` but does not):
```python
def append_bytes(self, obj: Iterable[Union[str, None]]) -> CArrayBuilder:
cdef Py_buffer buffer
cdef ArrowBufferView item
for py_item in obj:
if py_item is None:
code = ArrowArrayAppendNull(self._ptr, 1)
else:
PyObject_GetBuffer(py_item, &buffer, PyBUF_ANY_CONTIGUOUS | PyBUF_FORMAT)
if buffer.ndim != 1:
raise ValueError("Can't append buffer with dimensions != 1 to binary array")
if buffer.itemsize != 1:
PyBuffer_Release(&buffer)
raise ValueError("Can't append buffer with itemsize != 1 to binary array")
item.data.data = buffer.buf
item.size_bytes = buffer.len
code = ArrowArrayAppendBytes(self._ptr, item)
PyBuffer_Release(&buffer)
if code != NANOARROW_OK:
Error.raise_error(f"append bytes item {py_item}", code)
```
in nanobind, an equivalent implementation probably looks like this (N.B. this is untested):
```c++
auto AppendBytes(const CArrayBuilder &builder, nb::iterable obj)
ArrowErrorCode code;
ArrowBufferView item
for (const auto &py_item : obj) {
if (py_item.is_none()) {
code = ArrowArrayAppendNull(ptr_, 1);
} else {
Py_buffer buffer;
PyObject_GetBuffer(py_item, &buffer, PyBUF_ANY_CONTIGUOUS | PyBUF_FORMAT);
if (buffer.ndim != 1)
throw nb::value_error("Can't append buffer with dimensions != 1 to binary array");
if (buffer.itemsize != 1) {
PyBuffer_Release(&buffer);
throw nb::value_error("Can't append buffer with itemsize != 1 to binary array");
}
item.data.data = buffer.buf;
item.size_bytes = buffer.len;
code = ArrowArrayAppendBytes(ptr_, item);
PyBuffer_Release(&buffer);
}
if (code != NANOARROW_OK)
// TODO: would need to define NanoarrowException class as subclass of RuntimeError
throw NanoarrowException("append bytes item " + py_item.str() + " failed " + std::to_string(code));
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
This is a discussion rather than an implementation task, and it names no files or tests. Start by reviewing the current CArrayBuilder.append_bytes implementation and the Cython-generated nanoarrow headers/build setup; a concrete migration scope and acceptance criteria would be needed before defining what done means.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- build-system, developer-experience
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100