apache / apache/arrow-nanoarrow

Allow custom allocator to be injected in more places

Open
#690 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

When developing https://github.com/paleolimbot/duckdb-nanoarrow it was apparent that we need the ability to inject an `ArrowBufferAllocator` into the `ArrowIpcArrayStreamReader` for clients that need tight control over memory usage.

There is also at least one place in the device API where we assume a buffer allocator:

- https://github.com/apache/arrow-nanoarrow/blob/e54b7df525fa1d310a96687bd99902823402b26c/src/nanoarrow/nanoarrow_device.h#L220-L223

In those two examples specifically a lack of custom allocator isn't blocking any real-world case (in DuckDB we can also do a better job coordinating IO if we skip the `ArrowIpcArrayStreamReader` entirely and go through the lower-level `ArrowIpcDecoder`; in cudf they have their own code for copying arrays and don't use the device API); however, the option should still be there.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting ArrowIpcArrayStreamReader and the allocator assumptions at nanoarrow_device.h lines 220-223, then compare their use with the lower-level ArrowIpcDecoder. Done means clients can provide an ArrowBufferAllocator in both identified areas without relying on an implicit allocator.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.