Improving the CSV schema inference
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
There's an open issue in the datafusion repository with the CSV schema inference. The current implementation in arrow will return `Int64` as the datatype for any numeric columns that have no decimal and don't match a date format. This circumstance is causing problems when the CSV is read later, should the value overflow the `Int64` data type.
Here's the datafusion issue apache/arrow-datafusion#3174
**Describe the solution you'd like**
Maybe arrow could try to support the `UInt64` and `Decimal128` datatypes as well, should it notice the values inside the CSV are too large. Or even default to `String` should it notice that even these are too small to ensure the CSV can be read without problems.
**Describe alternatives you've considered**
Alternatively, I imagine the column's type could be "upgraded" when reading the CSV, should there be any parsing errors due to overflows. I imagine this would need all previously parsed values to be casted, which could hopefully be avoided given better inference results.
**Additional context**
I'd be open to implementing this change. My naive approach would be something like this: https://github.com/apache/arrow-rs/commit/4b3104ea431835018c4fb90003013e7d2c7fe47b# in case anyone here has any suggestions on how to improve it, I would be very happy.
Contributor guide
Research direction
Start by locating the CSV schema inference implementation in arrow-rs and review the linked DataFusion issue and referenced commit. Compare how numeric values are currently classified, then define the supported overflow-safe inference behavior and add coverage for values beyond Int64; done means CSV columns no longer infer an unsafe type for those inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100