User Defined Coercion Rules
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Coercion is (TODO find definition)
At the moment DataFusion has one set of built in coercion rules. However, with a single set of coercion rules, we'll always end up with conflicting requirements such as
- https://github.com/apache/datafusion/pull/14223
- https://github.com/apache/datafusion/issues/14272
- https://github.com/apache/datafusion/issues/14230 from @shehabgamin
It also makes it hard, as @findepi has pointed out several times, to extend DataFusion with new data types / logica types.
My conclusion is we will never be able to have the behavior that works for everyone, and any change in coercion is likely to make some other tradeoff.
Rather than having to go back/forth in the core, I think an API would give us an escape hatch.
### Describe the solution you'd like
While the user can in theory supply their own coercion rules by adding a new `AnalyzerRule` instead of the current [`TypeCoercion`] rule https://github.com/apache/datafusion/blob/18f14abf3fcef11f014f42e1f0cb8cdc35f257a3/datafusion/optimizer/src/analyzer/type_coercion.rs#L62 I think the coercion code is called in so many places this would only be a partial fix
Is it a crazy idea if we tried to implement "user defined coercion rules" -- that way we can have whatever coercion rules are in the DataFusion core but when they inevitably are not quite what users want the users can override them
It would also force us to design the type coercion APIs more coherently which I think would be good for the code quality in general
### Describe alternatives you've considered
I was imagining something like
```rust
struct MyCoercer {}
/// Implement custom coercion ehre
impl TypeCoercion for MyCoercer {
...
}
// register coercion rules like
let ctx: SessionContext = SessionStateBuilder::new()
.with_type_coercion_rules(Arc::new(MyCoercer{}));
...
```
The trait might look like
```rust
/// Defines coercion rules
pub trait TypeCoercion {
/// Given the types of arguments to a comparison operation (`=`, `<`, etc) what single type should all arguments be cast to
fn coerce_comparison(args: &[DataType]) -> DataType ;
...
}
```
Maybe there should be methods for `Field` instead of `DataType` 🤔
### Additional context
_No response_
Contributor guide
Research direction
Start with datafusion/optimizer/src/analyzer/type_coercion.rs and the TypeCoercion rule mentioned in the issue. Trace where coercion is called beyond the AnalyzerRule, then inspect SessionStateBuilder and the proposed with_type_coercion_rules entry point. Done means the API design and its integration points are defined, with the relevant coercion behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100