apache / apache/arrow

[Python] Improve error message when all values in a column are null in a parquet partition

Open
#32,955 7 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

There is a good bug report about this in and it still seems to be a problem.

Basically the error message is pretty bad if all values in a given column of a parquet partition are null. We should either handle this case better or give a better error message.

**Reporter**: [Philipp Moritz](https://issues.apache.org/jira/browse/ARROW-17719) / @pcmoritz

**Note**: *This issue was originally created as [ARROW-17719](https://issues.apache.org/jira/browse/ARROW-17719). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the all-null-column parquet partition case described in the linked Stack Overflow report, then trace the Python parquet-reading path responsible for the current error. Done means the case is handled appropriately or produces a clear, useful error message, with coverage for the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.