apache / apache/arrow-nanoarrow
[R] Implement ALTREP conversions for a list of arrays
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 12
Description
First noted in #66, it's fairly common to attempt a conversion of a stream with more than one batch to a data.frame. Currently this will convert one chunk at a time and `rbind()` or `c()` everything together. This is slow and requires at least twice the memory.
Related is the the "fixed size" converter path, which does a "preallocate + fill"; however this requires knowing the exact size before starting to pull batches which is almost never the case. The first bit could be solved by implementing the requisite copying functions to allow the pre-allocated vectors to be growable; however, that wouldn't allow for the individual components to be ALTREP...everything would be fully materialized.
The Arrow package handles this by a rather complicated implementation that has excellent type coverage: most chunked arrays can be wrapped in an ALTREP vector. Because we don't have Arrow C++ at our disposal, this is not practical here.
Somewhere in the middle is implementing a generic ALTREP vector of a concatenation: The "data" would be a `list()` of type-checked vectors (that could themselves be ALTREP); the ALTREP class would implement element access using something like the `ChunkResolver` sitting in Arrow C++.
Independently of that, implementing ALTREP conversion for a single `ArrowArray` -- particularly the ones that can share memory like int32/double with no nulls -- would reduce another copy. For types that can't share memory, lazily converting via the `ArrowArrayViewGet()` functions is also an option.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the existing stream conversion and fixed-size converter paths, then compare the proposed generic ALTREP concatenation with Arrow's approach and the ChunkResolver concept. Investigate ALTREP access for ArrowArray values, including ArrowArrayViewGet() and memory-sharing int32/double cases. Done means multi-batch conversion avoids materializing each chunk and supports the proposed single-ArrowArray conversions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100