[EPIC] Support Decimal for User Defined Functions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Math UDFs in the [datafusion-functions crate](https://github.com/apache/datafusion/tree/main/datafusion/functions) support integer and floating types, but not Decimals. This epic is dedicated to adding and improving decimal support for UDFs. It is a follow-up epic to adding Decimal support to the DataFusion core #3523.
So far, it is implemented for `log`, `power` binary UDFs; `round`, `ceil` unary UDF. Turns out, more code should be moved into helper functions to make UDFs leaner and abstracted from details (e.g., scalar vs. array cases, casting, etc).
#### Latest state (as for Aug 25)
Legend:
- :white_check_mark: - all good, implemented
- :x: - not supported or not applicable
- :white_large_square: - DataFusion implementation conforms to possible support, nothing to follow-up
| UDF | Is implemented | Could Support Decimal | Comments |
| -------------------------------- | ------------------------------------- | -------------------------------- | ----------------------------- |
| `abs` | :white_check_mark: | :white_check_mark: | Native support, #17808 |
| `any_value` | :white_check_mark: | :white_check_mark: | Native support by design |
| `approx_distinct` | :white_check_mark: | :white_check_mark: | Native support, #23190 |
| `approx_median` | :white_large_square: | :x: | Coerce to floats as with majority of DBs, #21074 |
| `approx_percentile_cont` | :white_large_square: | :x: | See `approx_median` |
| `approx_percentile_cont_with_weight` | :white_check_mark: | :white_check_mark: | See `approx_median` |
| `array_agg` | :white_check_mark: | :white_check_mark: | Native support by design |
| `avg` | :white_check_mark: | :white_check_mark: | Native support, #22713 |
| `bit_and` / `bit_or` / `bit_xor` | :white_large_square: | :x: | N/A, bit domain |
| `bool_and` / `bool_or` | :white_large_square: | :x: | N/A, boolean domain |
| `ceil` | :white_check_mark: | :white_check_mark: | Native support, #18979 |
| `corr` | :x: | :white_check_mark: | Coerced. Native support is hard. #3572 , #3481, #19463 |
| `count` / `count_distinct` | :white_large_square: | :x: | N/A, integer domain |
| `covar_samp` / `covar_pop` | :white_large_square: | :white_check_mark: | See `corr` |
| `degrees`,`radians`,`cot` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `exp` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `factorial` | :white_large_square: | :x: | N/A, integer domain |
| `first_value` / `last_value` | :white_check_mark: | :white_check_mark: | Native support, #17501 |
| `floor` | :white_check_mark: | :white_check_mark: | Native support, #18979 |
| `gcd` | :white_check_mark: | :white_check_mark: | Native support, #22655 |
| `greatest` | :white_check_mark: | :white_check_mark: | Native support by design |
| `isnan` | :white_check_mark: | :white_check_mark: | Native support, #20093 |
| `iszero` | :white_check_mark: | :white_check_mark: | Native support, #20093 |
| `least` | :white_check_mark: | :white_check_mark: | Native support by design |
| `lcm` | :white_check_mark: | :white_check_mark: | Native support, #22655 |
| `median` | :white_check_mark: | :white_check_mark: | Native support, #24419 |
| `log` | :white_check_mark: | :white_check_mark: | Partial support, #17023, #17555 |
| `ln`,`log2`,`log10` | :white_large_square: | :x: | N/A, float domain |
| `min` / `max` | :white_check_mark: | :white_check_mark: | Native support, #17501 |
| `nanvl` | :white_large_square: | :x: | N/A, decimals are not nans |
| `nth_value` | :white_check_mark: | :white_check_mark: | Native support by design |
| `percentile_cont` | :white_check_mark: | :white_check_mark: | Native support, #24419 |
| `pi`, `random` | :white_large_square: | :x: |N/A, float domain|
| `power` / `pow` | :white_large_square: | :x: | Decimals removed due to complexity, now coerced to floats. #18032 |
| `regr_*` (e.g. `slope`) | :white_large_square: | :x: | See `corr` |
| `round` | :white_check_mark: | :white_check_mark: | Native support, #17054, |
| `signum` | :white_large_square: | :white_check_mark: | N/A, returns integer |
| `sqrt`,`cbrt` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `stddev` / `stddev_pop` | :x: | :white_check_mark: | Coerced. Native support is hard. #3572 , #21926 |
| `sum` | :white_check_mark: | :white_check_mark: | Native support, #17591 |
| `trunc` | :white_check_mark: | :white_check_mark: | Native support, #23320 |
| `variance` / `var_pop` | :white_large_square: | :white_check_mark: | See `stddev` |
| trigonometric functions | :white_large_square: | :x: | N/A, float domain, coerced to float |
### Describe the solution you'd like
There are the following primary directions:
1) Adding support for well-known `Decimal128` and `Decimal256` to existing functions
2) Adding support for new `Decimal32` and `Decimal64`, which are not yet fully supported
3) Refining coercion rules to work with mixtures of floats/decimals
4) Ensuring it would work properly with the new `parse_float_as_decimal` flag, forcing floats to be decimals after SQL parsing
5) Improving tests to validate correct behaviour for floats/decimals and corner cases
6) Moving some core support to the Arrow libraries
I welcome thoughts and discussions about these directions.
### Describe alternatives you've considered
The approach of coercing decimals to floats could work, but it loses precision and data and doesn't match the behaviour of existing SQL engines (Postgres, Spark). Decimals should be first-class citizens.
### Additional context
**Related tickets:**
Function support:
- https://github.com/apache/datafusion/issues/17054
- https://github.com/apache/datafusion/issues/18031
- https://github.com/apache/datafusion/issues/17555
- https://github.com/apache/datafusion/issues/7689
- https://github.com/apache/datafusion/issues/18524
- https://github.com/apache/datafusion/issues/19250
- https://github.com/apache/datafusion/issues/19347
- https://github.com/apache/datafusion/issues/19348
- https://github.com/apache/datafusion/issues/19057
- https://github.com/apache/datafusion/issues/22512
- https://github.com/apache/datafusion/issues/22472
- https://github.com/apache/datafusion/issues/19921
- https://github.com/apache/datafusion/issues/20080
- https://github.com/apache/datafusion/issues/19536
- https://github.com/apache/datafusion/issues/1545
- https://github.com/apache/datafusion/issues/3572
- https://github.com/apache/datafusion/issues/22042
- https://github.com/apache/datafusion/issues/24576
- https://github.com/apache/datafusion/issues/24369
- https://github.com/apache/datafusion/issues/22511
- https://github.com/apache/datafusion/issues/20640
Core support:
- https://github.com/apache/datafusion/issues/18092
- https://github.com/apache/datafusion/issues/17489
- https://github.com/apache/datafusion/issues/17747
- https://github.com/apache/datafusion/issues/19621
- https://github.com/apache/datafusion/issues/21779
- https://github.com/apache/datafusion/issues/23835
Coercion and type issues:
- https://github.com/apache/datafusion/issues/14272
- https://github.com/apache/datafusion/issues/8795
- https://github.com/apache/datafusion/issues/16667
- https://github.com/apache/datafusion/issues/23095
Related but excluded from this epic:
- https://github.com/apache/datafusion/issues/14612
- https://github.com/apache/datafusion/issues/14760
- https://github.com/apache/datafusion/issues/14763
- https://github.com/apache/datafusion/issues/19004
Contributor guide
Research direction
Start by reviewing the datafusion-functions crate and the linked function-support, core-support, and coercion issues. Compare the listed UDF status table with the six proposed directions, then identify a single scoped change and its affected tests. Done should be defined as one agreed decimal-support improvement with passing coverage for the relevant float, decimal, and corner-case behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100