apache / apache/arrow-nanoarrow

[R] Implement ALTREP conversions for a list of arrays

Open
#219 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 16h
Merged PRs (30d)
12

Description

First noted in #66, it's fairly common to attempt a conversion of a stream with more than one batch to a data.frame. Currently this will convert one chunk at a time and `rbind()` or `c()` everything together. This is slow and requires at least twice the memory.

Related is the the "fixed size" converter path, which does a "preallocate + fill"; however this requires knowing the exact size before starting to pull batches which is almost never the case. The first bit could be solved by implementing the requisite copying functions to allow the pre-allocated vectors to be growable; however, that wouldn't allow for the individual components to be ALTREP...everything would be fully materialized.

The Arrow package handles this by a rather complicated implementation that has excellent type coverage: most chunked arrays can be wrapped in an ALTREP vector. Because we don't have Arrow C++ at our disposal, this is not practical here.

Somewhere in the middle is implementing a generic ALTREP vector of a concatenation: The "data" would be a `list()` of type-checked vectors (that could themselves be ALTREP); the ALTREP class would implement element access using something like the `ChunkResolver` sitting in Arrow C++.

Independently of that, implementing ALTREP conversion for a single `ArrowArray` -- particularly the ones that can share memory like int32/double with no nulls -- would reduce another copy. For types that can't share memory, lazily converting via the `ArrowArrayViewGet()` functions is also an option.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing stream conversion and fixed-size converter paths, then compare the proposed generic ALTREP concatenation with Arrow's approach and the ChunkResolver concept. Investigate ALTREP access for ArrowArray values, including ArrowArrayViewGet() and memory-sharing int32/double cases. Done means multi-batch conversion avoids materializing each chunk and supports the proposed single-ArrowArray conversions.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.