apache / apache/arrow-rs

Decimal enhancements in arrow-cast

Open
#5,068 4 comments 0 reactions 1 assignee Claimed by @neilconway View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
There are currently two different functions in arrow-cast which convert strings to decimals. `parse_string_to_decimal_native` is used by arrow-cast when casting an entire array from a string type to a decimal type. `parse_decimal` is used by arrow-csv and arrow-json. The inputs differ a bit, but the return types are identical. The latter accepts an additional precision argument, however when the former is used it is followed up by a call to `validate_decimal_precision` for the desired type. The former rounds values when input exceeds scale while the latter truncates. Neither function supports parsing scientific notation.

**Describe the solution you'd like**
1. Consolidate into a single, consistent function
2. Support scientific notation input
3. Default to "half up" rounding
4. Stretch goal: allow for configurable rounding (maybe as a CastOption?)

**Describe alternatives you've considered**
N/A

**Additional context**
I stumbled upon this when using arrow-json reader to build a record batch. I was getting errors when my decimal values contained scientific notation and noticed `parse_decimal` doesn't support it, despite scientific notation being valid for JSON numbers. When I tried forcing expanded notation I found the excess scale was being truncated instead of rounded. I then tried building my column using a string array and casting that to a decimal type. Instead of erroring on scientific notation the value was being treated as `null`. Expanded values did work and were rounded.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.