[C++][Python] The ORC writer does not support all-null object columns from Pandas
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Object columns in Pandas are currently converted to Null type columns in Arrow and they can not be written using `pyarrow.orc.write_table`. Since ORC doesn't support Null type columns we need to find the appropriate DataType for such columns and do the write.
**Reporter**: [Ian Alexander Joiner](https://issues.apache.org/jira/browse/ARROW-14782) / @iajoiner
**Note**: *This issue was originally created as [ARROW-14782](https://issues.apache.org/jira/browse/ARROW-14782). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start at pyarrow.orc.write_table and trace how Pandas object columns become Arrow Null type columns before reaching the ORC writer. Determine an appropriate DataType for all-null object columns and verify that such columns can be written successfully without the ORC Null type limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100