apache / apache/arrow

How to define that a field can be null

Open
#35,441 13 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

When defining a model using pa.schema, how do you define a field if the entire column may be empty?

For example, I define a name field, which may or may not be empty in the entire column
The definition is as follows:
```pa.field('name', pa.string())```

When the name column is empty, an error is reported, how do I define field so that it is compatible with the case where the whole column is empty

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported case using the pa.schema and pa.field calls shown in the issue, with a column that is empty throughout. Check the Python and Parquet schema documentation for the nullability behavior; done means the supported definition and the cause of the reported error are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.