apache / apache/datafusion

[EPIC] Support Decimal for User Defined Functions

Open
#18,889 2 comments 3 reactions 0 assignees View on GitHub
enhancement EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Math UDFs in the [datafusion-functions crate](https://github.com/apache/datafusion/tree/main/datafusion/functions) support integer and floating types, but not Decimals. This epic is dedicated to adding and improving decimal support for UDFs. It is a follow-up epic to adding Decimal support to the DataFusion core #3523.

So far, it is implemented for `log`, `power` binary UDFs; `round`, `ceil` unary UDF. Turns out, more code should be moved into helper functions to make UDFs leaner and abstracted from details (e.g., scalar vs. array cases, casting, etc).

#### Latest state (as for Aug 25)

Legend:
- :white_check_mark: - all good, implemented
- :x: - not supported or not applicable
- :white_large_square: - DataFusion implementation conforms to possible support, nothing to follow-up

| UDF | Is implemented | Could Support Decimal | Comments |
| -------------------------------- | ------------------------------------- | -------------------------------- | ----------------------------- |
| `abs` | :white_check_mark: | :white_check_mark: | Native support, #17808 |
| `any_value` | :white_check_mark: | :white_check_mark: | Native support by design |
| `approx_distinct` | :white_check_mark: | :white_check_mark: | Native support, #23190 |
| `approx_median` | :white_large_square: | :x: | Coerce to floats as with majority of DBs, #21074 |
| `approx_percentile_cont` | :white_large_square: | :x: | See `approx_median` |
| `approx_percentile_cont_with_weight` | :white_check_mark: | :white_check_mark: | See `approx_median` |
| `array_agg` | :white_check_mark: | :white_check_mark: | Native support by design |
| `avg` | :white_check_mark: | :white_check_mark: | Native support, #22713 |
| `bit_and` / `bit_or` / `bit_xor` | :white_large_square: | :x: | N/A, bit domain |
| `bool_and` / `bool_or` | :white_large_square: | :x: | N/A, boolean domain |
| `ceil` | :white_check_mark: | :white_check_mark: | Native support, #18979 |
| `corr` | :x: | :white_check_mark: | Coerced. Native support is hard. #3572 , #3481, #19463 |
| `count` / `count_distinct` | :white_large_square: | :x: | N/A, integer domain |
| `covar_samp` / `covar_pop` | :white_large_square: | :white_check_mark: | See `corr` |
| `degrees`,`radians`,`cot` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `exp` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `factorial` | :white_large_square: | :x: | N/A, integer domain |
| `first_value` / `last_value` | :white_check_mark: | :white_check_mark: | Native support, #17501 |
| `floor` | :white_check_mark: | :white_check_mark: | Native support, #18979 |
| `gcd` | :white_check_mark: | :white_check_mark: | Native support, #22655 |
| `greatest` | :white_check_mark: | :white_check_mark: | Native support by design |
| `isnan` | :white_check_mark: | :white_check_mark: | Native support, #20093 |
| `iszero` | :white_check_mark: | :white_check_mark: | Native support, #20093 |
| `least` | :white_check_mark: | :white_check_mark: | Native support by design |
| `lcm` | :white_check_mark: | :white_check_mark: | Native support, #22655 |
| `median` | :white_check_mark: | :white_check_mark: | Native support, #24419 |
| `log` | :white_check_mark: | :white_check_mark: | Partial support, #17023, #17555 |
| `ln`,`log2`,`log10` | :white_large_square: | :x: | N/A, float domain |
| `min` / `max` | :white_check_mark: | :white_check_mark: | Native support, #17501 |
| `nanvl` | :white_large_square: | :x: | N/A, decimals are not nans |
| `nth_value` | :white_check_mark: | :white_check_mark: | Native support by design |
| `percentile_cont` | :white_check_mark: | :white_check_mark: | Native support, #24419 |
| `pi`, `random` | :white_large_square: | :x: |N/A, float domain|
| `power` / `pow` | :white_large_square: | :x: | Decimals removed due to complexity, now coerced to floats. #18032 |
| `regr_*` (e.g. `slope`) | :white_large_square: | :x: | See `corr` |
| `round` | :white_check_mark: | :white_check_mark: | Native support, #17054, |
| `signum` | :white_large_square: | :white_check_mark: | N/A, returns integer |
| `sqrt`,`cbrt` | :white_check_mark: | :x: | N/A, float domain, coerced to float |
| `stddev` / `stddev_pop` | :x: | :white_check_mark: | Coerced. Native support is hard. #3572 , #21926 |
| `sum` | :white_check_mark: | :white_check_mark: | Native support, #17591 |
| `trunc` | :white_check_mark: | :white_check_mark: | Native support, #23320 |
| `variance` / `var_pop` | :white_large_square: | :white_check_mark: | See `stddev` |
| trigonometric functions | :white_large_square: | :x: | N/A, float domain, coerced to float |

### Describe the solution you'd like

There are the following primary directions:
1) Adding support for well-known `Decimal128` and `Decimal256` to existing functions
2) Adding support for new `Decimal32` and `Decimal64`, which are not yet fully supported
3) Refining coercion rules to work with mixtures of floats/decimals
4) Ensuring it would work properly with the new `parse_float_as_decimal` flag, forcing floats to be decimals after SQL parsing
5) Improving tests to validate correct behaviour for floats/decimals and corner cases
6) Moving some core support to the Arrow libraries

I welcome thoughts and discussions about these directions.

### Describe alternatives you've considered

The approach of coercing decimals to floats could work, but it loses precision and data and doesn't match the behaviour of existing SQL engines (Postgres, Spark). Decimals should be first-class citizens.

### Additional context

**Related tickets:**

Function support:
- https://github.com/apache/datafusion/issues/17054
- https://github.com/apache/datafusion/issues/18031
- https://github.com/apache/datafusion/issues/17555
- https://github.com/apache/datafusion/issues/7689
- https://github.com/apache/datafusion/issues/18524
- https://github.com/apache/datafusion/issues/19250
- https://github.com/apache/datafusion/issues/19347
- https://github.com/apache/datafusion/issues/19348
- https://github.com/apache/datafusion/issues/19057
- https://github.com/apache/datafusion/issues/22512
- https://github.com/apache/datafusion/issues/22472
- https://github.com/apache/datafusion/issues/19921
- https://github.com/apache/datafusion/issues/20080
- https://github.com/apache/datafusion/issues/19536
- https://github.com/apache/datafusion/issues/1545
- https://github.com/apache/datafusion/issues/3572
- https://github.com/apache/datafusion/issues/22042
- https://github.com/apache/datafusion/issues/24576
- https://github.com/apache/datafusion/issues/24369
- https://github.com/apache/datafusion/issues/22511
- https://github.com/apache/datafusion/issues/20640

Core support:
- https://github.com/apache/datafusion/issues/18092
- https://github.com/apache/datafusion/issues/17489
- https://github.com/apache/datafusion/issues/17747
- https://github.com/apache/datafusion/issues/19621
- https://github.com/apache/datafusion/issues/21779
- https://github.com/apache/datafusion/issues/23835

Coercion and type issues:
- https://github.com/apache/datafusion/issues/14272
- https://github.com/apache/datafusion/issues/8795
- https://github.com/apache/datafusion/issues/16667
- https://github.com/apache/datafusion/issues/23095

Related but excluded from this epic:
- https://github.com/apache/datafusion/issues/14612
- https://github.com/apache/datafusion/issues/14760
- https://github.com/apache/datafusion/issues/14763
- https://github.com/apache/datafusion/issues/19004

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the datafusion-functions crate and the linked function-support, core-support, and coercion issues. Compare the listed UDF status table with the six proposed directions, then identify a single scoped change and its affected tests. Done should be defined as one agreed decimal-support improvement with passing coverage for the relevant float, decimal, and corner-case behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.