[BUG] Special case Parquet LIST names appear to be ignored
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
The parquet specification at https://github.com/apache/parquet-format/blob/master/LogicalTypes.md when talking about backwards compatibility in lists says that
> If the repeated field is a group with one field and is named either array or uses the LIST-annotated group's name with _tuple appended then the repeated type is the element type and elements are required.
The examples given for these are.
```
// List> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated group array {
required binary str (UTF8);
};
}
// List> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated group my_list_tuple {
required binary str (UTF8);
};
}
```
I implemented some tests based off of this and saw the CUDF is able to parse the data, but it is not returning the same types as Spark does, nor does it return what I would expect the examples to show.
In [files.zip](https://github.com/rapidsai/cudf/files/9913952/files.zip) there are two parquet files.
`SPECIAL_ARRAY_LIST_TEST.parquet` has a footer schema of
```
message spark {
required group my_list (LIST) {
repeated group array {
required int32 item;
}
}
}
```
When I parse the data with CUDF I get back a table with types like `Table>`, but Spark and expects the data to look like `Table>>`.
Pandas appears to do the same thing, but I am not an expert on pandas to be 100% sure that it is the same thing.
```python
>>> pd.read_parquet("SPECIAL_ARRAY_LIST_TEST.parquet")
my_list
0 [{'item': 0}, {'item': 1}]
>>> pd.read_parquet("SPECIAL_ARRAY_LIST_TEST.parquet").info()
RangeIndex: 1 entries, 0 to 0
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 my_list 1 non-null object
dtypes: object(1)
memory usage: 136.0+ bytes
```
The other file is essentially the same, but it is using the `_tuple` special case instead of `array`.
**Steps/Code to reproduce bug**
Try to read the attached files in CUDF and see if they match the desired types/schema.
**Expected behavior**
They should match, but it looks like they do not.
**Additional context**
This is probably not super critical because it is an odd corner case that is not likely to be very common, but technically it is returning the wrong data.
Contributor guide
Assessment
This issue has not been assessed yet.