Decimal enhancements in arrow-cast
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
There are currently two different functions in arrow-cast which convert strings to decimals. `parse_string_to_decimal_native` is used by arrow-cast when casting an entire array from a string type to a decimal type. `parse_decimal` is used by arrow-csv and arrow-json. The inputs differ a bit, but the return types are identical. The latter accepts an additional precision argument, however when the former is used it is followed up by a call to `validate_decimal_precision` for the desired type. The former rounds values when input exceeds scale while the latter truncates. Neither function supports parsing scientific notation.
**Describe the solution you'd like**
1. Consolidate into a single, consistent function
2. Support scientific notation input
3. Default to "half up" rounding
4. Stretch goal: allow for configurable rounding (maybe as a CastOption?)
**Describe alternatives you've considered**
N/A
**Additional context**
I stumbled upon this when using arrow-json reader to build a record batch. I was getting errors when my decimal values contained scientific notation and noticed `parse_decimal` doesn't support it, despite scientific notation being valid for JSON numbers. When I tried forcing expanded notation I found the excess scale was being truncated instead of rounded. I then tried building my column using a string array and casting that to a decimal type. Instead of erroring on scientific notation the value was being treated as `null`. Expanded values did work and were rounded.
Contributor guide
Assessment
This issue has not been assessed yet.