apache / apache/arrow

[R] Selecting colums while reading Parquet file with nested types can give wrong column

Open
#18,882 1 comment 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I created two small files (using Python for my convenience):

```python

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({"a": [1, 2], "b": [3, 4]})
pq.write_table(table, "test1.parquet")

table = pa.table({"a": [1, 2], "nested": [[{'f1': 1, 'f2': 3}, {'f1': 3, 'f2': 4}], None], "b": [3, 4]})
pq.write_table(table, "test2.parquet")
```

where the first is a simple file, and the second contains a column with a nested list of struct type.

Reading that in R with a column selection works in the first case, but actually reads the second column instead of third in the second case:

```r

> arrow::read_parquet("test1.parquet", col_select=c("b"))
b
1 3
2 4
> arrow::read_parquet("test2.parquet", col_select=c("b"))
nested
1 3, 4
2 NULL
```

This is due to the simple conversion of column names to integer indices in the R code, while Parquet counts the individual fields of nested columns separately.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-14287) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-14287](https://issues.apache.org/jira/browse/ARROW-14287). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue by generating test1.parquet and test2.parquet with the Python examples, then run arrow::read_parquet with col_select=c("b"). Trace the R column-name-to-index conversion and compare it with Parquet's treatment of nested fields. Done means selecting b returns the b column for both files, with regression coverage for the nested case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.