apache / apache/arrow

[Python] Extend csv `invalid_row_handler` with `fill_null` and `ignore_unknown`

Open
#35,198 1 comment 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Referencing a past comment by @jorisvandenbossche,

_A bit orthogonal to this PR, but while looking at it: it could be an extension to allow an additional handler return value of "fill_null" (or another name), which could signal that in case the row has fewer fields, the missing fields can be filled with nulls (this is actually the default behaviour in pandas)._

_That's at least theoretically doable, perhaps you can open a separate JIRA for that?_

_Originally posted by @pitrou in https://github.com/apache/arrow/issues/12012#issuecomment-1002637825_

Along with adding `fill_null` value, it would be good "the opposite", `ignore_unknown` (or `ignore_value`) in case there are more unpacked values than columns in the schema.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Python CSV invalid_row_handler entry point and its existing tests. Read how handler return values are interpreted for rows with missing or extra fields. Done means both fill_null and ignore_unknown are supported and covered by tests for the respective row-shape cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.