apache / apache/datafusion

Alter `ScalarValue` APIs to be more clear when an operation isn't permitted

Open
#19,436 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Some of the APIs for `ScalarValue` are confusing in which operations are permitted. For example:

https://github.com/apache/datafusion/blob/5419ff5902dfb7a9a1099022fefe289bd5de71ec/datafusion/common/src/scalar/mod.rs#L1446-L1452

If the data type doesn't support the concept of a zero value, we return an error. Ideally callers would check for this error, but usually we just bubble any errors up with `?`. Perhaps we should consider changing the return to be an `Option` so the caller can match on whether the given datatype has a zero value or not, instead of resorting to using `Result` to represent this?

Another example:

https://github.com/apache/datafusion/blob/5419ff5902dfb7a9a1099022fefe289bd5de71ec/datafusion/common/src/scalar/mod.rs#L1798-L1801

Here, for decimals we can't actually represent a value of ten if we have negative scale less than -1; so we return an internal error in that case, but that means callers need to either prevent calling this function with a negative scale decimal (that isn't -1) or match on the result. If we encode it as `Result>` instead, with the `Result` reserved for returning errors in case of unusual behaviour (e.g. invalid precision/scale combination), it can make it more clear that sometimes the value of ten just doesn't exist.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/common/src/scalar/mod.rs at the cited zero-value API around lines 1446-1452 and decimal API around lines 1798-1801. Read their callers to understand how unsupported operations are currently propagated. Done means the API semantics clearly distinguish unsupported values from unexpected errors and affected callers are updated consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.