[Python] Extend csv `invalid_row_handler` with `fill_null` and `ignore_unknown`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Referencing a past comment by @jorisvandenbossche,
_A bit orthogonal to this PR, but while looking at it: it could be an extension to allow an additional handler return value of "fill_null" (or another name), which could signal that in case the row has fewer fields, the missing fields can be filled with nulls (this is actually the default behaviour in pandas)._
_That's at least theoretically doable, perhaps you can open a separate JIRA for that?_
_Originally posted by @pitrou in https://github.com/apache/arrow/issues/12012#issuecomment-1002637825_
Along with adding `fill_null` value, it would be good "the opposite", `ignore_unknown` (or `ignore_value`) in case there are more unpacked values than columns in the schema.
Contributor guide
Research direction
Start by locating the Python CSV invalid_row_handler entry point and its existing tests. Read how handler return values are interpreted for rows with missing or extra fields. Done means both fill_null and ignore_unknown are supported and covered by tests for the respective row-shape cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100