NVIDIA / NVIDIA/cudf

[BUG] Backwards compatible parquet MAP_KEY_VALUE is not treated properly

Open
#12,044 1 comment 0 reactions 0 assignees View on GitHub
0 - Backlog bug cuIO libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
The parquet specification at https://github.com/apache/parquet-format/blob/master/LogicalTypes.md when talking about backwards compatibility in maps says that

> Some existing data incorrectly used MAP_KEY_VALUE in place of MAP. For backward-compatibility, a group annotated with MAP_KEY_VALUE that is not contained by a MAP-annotated group should be handled as a MAP-annotated group.

The example schema given for this is.
```
// Map (nullable map, nullable values)
optional group my_map (MAP_KEY_VALUE) {
repeated group map {
required binary key (UTF8);
optional int32 value;
}
}
```

I created a parquet file and put it in [file.zip](https://github.com/rapidsai/cudf/files/9914136/file.zip) that is very similar, but it uses `int32` for both the key and the value.

```
message spark {
required group my_map (MAP_KEY_VALUE) {
repeated group map {
required int32 key;
required int32 value;
}
}
}
```

When I read the data back using CUDF I get a schema like `TABLE>>`, but what we want is `TABLE>>`. Because that first column is a STRUCT and not a LIST only the first row in the LIST is returned.

It looks like panads is able to do this.

```python
>>> pd.read_parquet("MAP_KEY_VALUE_TEST.parquet")
my_map
0 [(0, 2), (1, 3)]
>>> pd.read_parquet("MAP_KEY_VALUE_TEST.parquet").info()

RangeIndex: 1 entries, 0 to 0
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 my_map 1 non-null object
dtypes: object(1)
memory usage: 136.0+ bytes
```

**Additional context**
This is probably not a super high priority. It is an odd/rare corner case. At least until a customer hit this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.