apache / apache/arrow

[R] Factor levels cannot contain NA

Open
#32,102 6 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Medium Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

If you try to write a data frame with a factor with a missing value to parquet, you get the error: "Error: Invalid: Cannot insert dictionary values containing nulls". 

This seems likely due to how the metadata for factors is currently captured in parquet files. Reprex follows:

 

library(arrow)

bad_data <- data.frame(A = factor(1, 2, NA))

write_parquet(bad_data, tempfile())

 

 

**Reporter**: [Kieran Martin](https://issues.apache.org/jira/browse/ARROW-16768)

**Note**: *This issue was originally created as [ARROW-16768](https://issues.apache.org/jira/browse/ARROW-16768). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the provided R reprex with arrow's write_parquet() and inspect the factor metadata handling involved in writing Parquet data. Done means a data frame containing a factor with NA can be written without the dictionary-null error, with the expected missing value preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.