apache / apache/arrow

[C++][Python] The ORC writer does not support all-null object columns from Pandas

Open
#30,317 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Object columns in Pandas are currently converted to Null type columns in Arrow and they can not be written using `pyarrow.orc.write_table`. Since ORC doesn't support Null type columns we need to find the appropriate DataType for such columns and do the write.

**Reporter**: [Ian Alexander Joiner](https://issues.apache.org/jira/browse/ARROW-14782) / @iajoiner

**Note**: *This issue was originally created as [ARROW-14782](https://issues.apache.org/jira/browse/ARROW-14782). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start at pyarrow.orc.write_table and trace how Pandas object columns become Arrow Null type columns before reaching the ORC writer. Determine an appropriate DataType for all-null object columns and verify that such columns can be written successfully without the ORC Null type limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.