[R] Selecting colums while reading Parquet file with nested types can give wrong column
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
I created two small files (using Python for my convenience):
```python
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.table({"a": [1, 2], "b": [3, 4]})
pq.write_table(table, "test1.parquet")
table = pa.table({"a": [1, 2], "nested": [[{'f1': 1, 'f2': 3}, {'f1': 3, 'f2': 4}], None], "b": [3, 4]})
pq.write_table(table, "test2.parquet")
```
where the first is a simple file, and the second contains a column with a nested list of struct type.
Reading that in R with a column selection works in the first case, but actually reads the second column instead of third in the second case:
```r
> arrow::read_parquet("test1.parquet", col_select=c("b"))
b
1 3
2 4
> arrow::read_parquet("test2.parquet", col_select=c("b"))
nested
1 3, 4
2 NULL
```
This is due to the simple conversion of column names to integer indices in the R code, while Parquet counts the individual fields of nested columns separately.
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-14287) / @jorisvandenbossche
**Note**: *This issue was originally created as [ARROW-14287](https://issues.apache.org/jira/browse/ARROW-14287). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the issue by generating test1.parquet and test2.parquet with the Python examples, then run arrow::read_parquet with col_select=c("b"). Trace the R column-name-to-index conversion and compare it with Parquet's treatment of nested fields. Done means selecting b returns the b column for both files, with regression coverage for the nested case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100