apache / apache/arrow

[C++] Read list/array data from ChunkedArray with multiple chunks

Open
#33,481 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: test
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I am reading a parquet file with arrow::RecordBatchReader and the arrow::Table returned contains columns with multiple chunks (column->num_chunks() > 1). The column in question, although not limited to, is of type Array(Int64).

 

I want to convert this arrow column into an internal structure that contains a contiguous chunk of memory for the data and a vector of offsets, very similar to arrow's structure. The code I have so far works in two "phases":

1. Get nested arrow column data. In that case, get Int64 data out of Array(Int64).
2. Get offsets from Array(Int64).

To achieve the #1, I am looping over the chunks and storing arrow::Array::values into a new arrow::ChunkedArray.

 
```java

static std::shared_ptr getNestedArrowColumn(std::shared_ptr & arrow_column)
{
arrow::ArrayVector array_vector;
array_vector.reserve(arrow_column->num_chunks());
for (size_t chunk_i = 0, num_chunks = static_cast(arrow_column->num_chunks()); chunk_i < num_chunks; ++chunk_i)
{
arrow::ListArray & list_chunk = dynamic_cast(*(arrow_column->chunk(chunk_i)));
std::shared_ptr chunk = list_chunk.values();
array_vector.emplace_back(std::move(chunk));
}
return std::make_shared(array_vector);
}
```
This does not work as expected, tho. Even though there are multiple chunks, the arrow::Array::values method returns the very same buffer for all of them, which ends up duplicating the data on my side. One pattern I noticed is that if I read only the Array(Int64) column, I get only one chunk. If I read both columns, I get two chunks. It looks like all columns will, inevitably, have the same number of chunks, even though its buffer is not chunked accordingly.

I then looked through more examples and came across the [ColumnarTableToVector example](https://github.com/apache/arrow/blob/master/cpp/examples/arrow/row_wise_conversion_example.cc#L121). It looks like this example assumes there is only on chunk and ignores the possibility of it having multiple chunks. It's probably just a detail and the test wasn't actually intended to cover multiple chunks.

I managed to get the expected output doing something like the below:
```java

auto & list_chunk1 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(0)));
auto & list_chunk2 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(1)));

auto l1_offset = *list_chunk1.raw_value_offsets();
auto l2_offset = *list_chunk2.raw_value_offsets();

auto l1_end_offset = list_chunk1.value_offset(list_chunk1.data()->length);
auto l2_end_offset = list_chunk2.value_offset(list_chunk2.data()->length);

auto lcv1 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(0))).values()->SliceSafe(l1_offset, l1_end_offset - l1_offset).ValueOrDie();
auto lcv2 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(1))).values()->SliceSafe(l2_offset, l2_end_offset - l2_offset).ValueOrDie();
```
This looks too hackish and I feel like there is a much better way.

Hence, my question: How do I properly extract the data & offsets out of such column? A more generic version of this is: how to extract the data out of ChunkedArrays with multiple chunks?

**Reporter**: [Arthur Passos](https://issues.apache.org/jira/browse/ARROW-18307)

**Note**: *This issue was originally created as [ARROW-18307](https://issues.apache.org/jira/browse/ARROW-18307). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading cpp/examples/arrow/row_wise_conversion_example.cc around ColumnarTableToVector, then inspect the ChunkedArray and ListArray entry points mentioned in the report, including chunk(), values(), raw_value_offsets(), and SliceSafe(). Done should define a supported way to extract nested data and offsets when a column has multiple chunks, with coverage for the reported multi-column case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.