apache / apache/datafusion

User Defined Coercion Rules

Open
#14,296 14 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Coercion is (TODO find definition)

At the moment DataFusion has one set of built in coercion rules. However, with a single set of coercion rules, we'll always end up with conflicting requirements such as
- https://github.com/apache/datafusion/pull/14223
- https://github.com/apache/datafusion/issues/14272
- https://github.com/apache/datafusion/issues/14230 from @shehabgamin

It also makes it hard, as @findepi has pointed out several times, to extend DataFusion with new data types / logica types.

My conclusion is we will never be able to have the behavior that works for everyone, and any change in coercion is likely to make some other tradeoff.

Rather than having to go back/forth in the core, I think an API would give us an escape hatch.

### Describe the solution you'd like

While the user can in theory supply their own coercion rules by adding a new `AnalyzerRule` instead of the current [`TypeCoercion`] rule https://github.com/apache/datafusion/blob/18f14abf3fcef11f014f42e1f0cb8cdc35f257a3/datafusion/optimizer/src/analyzer/type_coercion.rs#L62 I think the coercion code is called in so many places this would only be a partial fix

Is it a crazy idea if we tried to implement "user defined coercion rules" -- that way we can have whatever coercion rules are in the DataFusion core but when they inevitably are not quite what users want the users can override them

It would also force us to design the type coercion APIs more coherently which I think would be good for the code quality in general

### Describe alternatives you've considered

I was imagining something like

```rust
struct MyCoercer {}

/// Implement custom coercion ehre
impl TypeCoercion for MyCoercer {
...
}

// register coercion rules like
let ctx: SessionContext = SessionStateBuilder::new()
.with_type_coercion_rules(Arc::new(MyCoercer{}));
...
```

The trait might look like

```rust
/// Defines coercion rules
pub trait TypeCoercion {
/// Given the types of arguments to a comparison operation (`=`, `<`, etc) what single type should all arguments be cast to
fn coerce_comparison(args: &[DataType]) -> DataType ;
...
}
```

Maybe there should be methods for `Field` instead of `DataType` 🤔

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/optimizer/src/analyzer/type_coercion.rs and the TypeCoercion rule mentioned in the issue. Trace where coercion is called beyond the AnalyzerRule, then inspect SessionStateBuilder and the proposed with_type_coercion_rules entry point. Done means the API design and its integration points are defined, with the relevant coercion behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.