apache / apache/arrow-nanoarrow
Allow custom allocator to be injected in more places
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 11
Description
When developing https://github.com/paleolimbot/duckdb-nanoarrow it was apparent that we need the ability to inject an `ArrowBufferAllocator` into the `ArrowIpcArrayStreamReader` for clients that need tight control over memory usage.
There is also at least one place in the device API where we assume a buffer allocator:
- https://github.com/apache/arrow-nanoarrow/blob/e54b7df525fa1d310a96687bd99902823402b26c/src/nanoarrow/nanoarrow_device.h#L220-L223
In those two examples specifically a lack of custom allocator isn't blocking any real-world case (in DuckDB we can also do a better job coordinating IO if we skip the `ArrowIpcArrayStreamReader` entirely and go through the lower-level `ArrowIpcDecoder`; in cudf they have their own code for copying arrays and don't use the device API); however, the option should still be there.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting ArrowIpcArrayStreamReader and the allocator assumptions at nanoarrow_device.h lines 220-223, then compare their use with the lower-level ArrowIpcDecoder. Done means clients can provide an ArrowBufferAllocator in both identified areas without relying on an implicit allocator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100