apache / apache/arrow

[C++][Parquet] Strange behavior when null is partition value in parquet datasets with filename and directory partitioning

Open
#39,938 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

```
null_table = pa.Table.from_pydict({"A": ["", None], "B": [None, ""], "C": ["E", "F"]})
partitioning = pa.dataset.FilenamePartitioning(pa.schema({"A": pa.string()}))
pa.parquet.write_to_dataset(null_table, r"dir\filename\", partitioning=partitioning)
>>> pa.parquet.read_table( r"dir\filename\", partitioning=partitioning)
pyarrow.Table
A: string
----
A: []
>>>
```

Null does not roundtrip when it is one of the partition values, and filename or directory partitioning are used.

When mixing null and empty string with directory partitioning, it also doesn't roundtrip:

```
>>> partitioning = pa.dataset.DirectoryPartitioning(pa.schema({"A": pa.string()}))
>>> pa.parquet.write_to_dataset(null_table, r"dir\directory\", partitioning=partitioning)
>>> pa.parquet.read_table( r"dir\directory\", partitioning=partitioning)
pyarrow.Table
B: string
C: string
A: string
----
B: [[null],[""]]
C: [["E"],["F"]]
A: [[null],[null]]
>>>
```

I would expect A to be:

`A: [[""],[null]]`

Which is what the original table had. Also, if there are 2 partition columns, and the first one has a null value, an error is raised, but if there's only 1 partition column, then no error is raised, which seems inconsistent:

```
>>> partitioning = pa.dataset.DirectoryPartitioning(pa.schema({"A": pa.string(), "B": pa.string()}))
>>> pa.parquet.write_to_dataset(null_table, r"dir\", partitioning=partitioning)
pyarrow.lib.ArrowInvalid: No partition key for A but a key was provided subsequently for B.
>>>
```

What is the expected behavior when null is in a partition column? Is it expected to work, or should an error always be raised?

### Component(s)

C++, Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.