apache / apache/arrow

[R] [parquet] Cannot load chunked large_list<dict> from parquet file

Open
#45,087 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The R arrow library cannot load a file with schema
```
schema: codes: large_list>
child 0, element: dictionary
```
if the table is chunked. To reproduce, run below python script. in an environment that also has R with arrow installed

```python
import pyarrow as pa
import pyarrow.parquet as pq
import subprocess

def test_load_parquet(table,label):
pq.write_table(table, "t.parquet", row_group_size=1)
res = subprocess.run(
["Rscript", "-e", 'library(arrow);t=arrow::read_parquet("t.parquet");'],
capture_output=True,
)
print(f'{label}\n#####')
if res.returncode != 0:
stdErr = res.stderr.decode()
assert "NotImplemented: Nested data conversions not implemented for chunked array outputs" in stdErr
print('R failed')
else:
print('R ok')

pq.read_table("t.parquet") # no error!
print("python ok")
print("schema:",pq.read_schema("t.parquet"))

codes = [["a"],["a"]]
t1 = pa.table({"codes": codes})
t2 = pa.table({"codes": codes}).cast(
pa.schema({"codes": pa.large_list(pa.dictionary(pa.int32(), pa.string()))})
)
t3 = pa.table({"codes": codes}).cast(
pa.schema({"codes": pa.list_(pa.dictionary(pa.int32(), pa.string()))})
)
test_load_parquet(t1,'t1')
test_load_parquet(t2,'t2')
test_load_parquet(t3,'t3')

```

to get the output
```
t1
#####
R ok
python ok
schema: codes: list
child 0, element: string
t2
#####
R failed
python ok
schema: codes: large_list>
child 0, element: dictionary
t3
#####
R failed
python ok
schema: codes: list>
child 0, element: dictionary
```

I have verified this is an issue in R library versions 13.0.0.0 and 18.1.0. both list_ and large_list fails.

The error reported by the R library is discussed in #32723 , but since this works in pyarrow, I guess this is a separate issue from the C++ issue.

### Component(s)

Parquet, R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.