apache / apache/arrow

[C++][Python] Add default data type for completely null arrays when field is not populated in pyarrow.csv.ConvertOptions

Open
#47,314 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Status: stale-warning Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

I receive regularly updated csv files from a Government source. Sometimes a particular field is not populated in a supply.

I'd like to store as parquet and keep schemas consistent.

As a workaround, I check the schemas and case null arrays as strings, but it would be good to have something like:

```null_values_type = pa.String()```

Added to ConvertOptions. Thank you.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.