[C++] Read list/array data from ChunkedArray with multiple chunks
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
I am reading a parquet file with arrow::RecordBatchReader and the arrow::Table returned contains columns with multiple chunks (column->num_chunks() > 1). The column in question, although not limited to, is of type Array(Int64).
I want to convert this arrow column into an internal structure that contains a contiguous chunk of memory for the data and a vector of offsets, very similar to arrow's structure. The code I have so far works in two "phases":
1. Get nested arrow column data. In that case, get Int64 data out of Array(Int64).
2. Get offsets from Array(Int64).
To achieve the #1, I am looping over the chunks and storing arrow::Array::values into a new arrow::ChunkedArray.
```java
static std::shared_ptr getNestedArrowColumn(std::shared_ptr & arrow_column)
{
arrow::ArrayVector array_vector;
array_vector.reserve(arrow_column->num_chunks());
for (size_t chunk_i = 0, num_chunks = static_cast(arrow_column->num_chunks()); chunk_i < num_chunks; ++chunk_i)
{
arrow::ListArray & list_chunk = dynamic_cast(*(arrow_column->chunk(chunk_i)));
std::shared_ptr chunk = list_chunk.values();
array_vector.emplace_back(std::move(chunk));
}
return std::make_shared(array_vector);
}
```
This does not work as expected, tho. Even though there are multiple chunks, the arrow::Array::values method returns the very same buffer for all of them, which ends up duplicating the data on my side. One pattern I noticed is that if I read only the Array(Int64) column, I get only one chunk. If I read both columns, I get two chunks. It looks like all columns will, inevitably, have the same number of chunks, even though its buffer is not chunked accordingly.
I then looked through more examples and came across the [ColumnarTableToVector example](https://github.com/apache/arrow/blob/master/cpp/examples/arrow/row_wise_conversion_example.cc#L121). It looks like this example assumes there is only on chunk and ignores the possibility of it having multiple chunks. It's probably just a detail and the test wasn't actually intended to cover multiple chunks.
I managed to get the expected output doing something like the below:
```java
auto & list_chunk1 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(0)));
auto & list_chunk2 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(1)));
auto l1_offset = *list_chunk1.raw_value_offsets();
auto l2_offset = *list_chunk2.raw_value_offsets();
auto l1_end_offset = list_chunk1.value_offset(list_chunk1.data()->length);
auto l2_end_offset = list_chunk2.value_offset(list_chunk2.data()->length);
auto lcv1 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(0))).values()->SliceSafe(l1_offset, l1_end_offset - l1_offset).ValueOrDie();
auto lcv2 = dynamic_cast<::arrow::ListArray &>(*(arrow_column->chunk(1))).values()->SliceSafe(l2_offset, l2_end_offset - l2_offset).ValueOrDie();
```
This looks too hackish and I feel like there is a much better way.
Hence, my question: How do I properly extract the data & offsets out of such column? A more generic version of this is: how to extract the data out of ChunkedArrays with multiple chunks?
**Reporter**: [Arthur Passos](https://issues.apache.org/jira/browse/ARROW-18307)
**Note**: *This issue was originally created as [ARROW-18307](https://issues.apache.org/jira/browse/ARROW-18307). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reading cpp/examples/arrow/row_wise_conversion_example.cc around ColumnarTableToVector, then inspect the ChunkedArray and ListArray entry points mentioned in the report, including chunk(), values(), raw_value_offsets(), and SliceSafe(). Done should define a supported way to extract nested data and offsets when a column has multiple chunks, with coverage for the reported multi-column case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100