Should `PruningPredicate` coerce?
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Currently when you pass a pruning predicate where the predicate has a different type as the targeted column it will not prune it, even though in theory the value is castable to the target column type.
The predicate looks like this: "month_id = '202502' AND date_id = '20250226'"
However the columns are `int` column not `utf8`. So in theory these string values can be casted to int but I don't believe this is happening. Is this something that should be added
This is our rust code btw
```
// delta-rs/crates/core/src/delta_datafusion/mod.rs
let pruning_predicate = PruningPredicate::try_new(predicate.clone(), logical_schema.clone())?;
let files_to_prune = pruning_predicate.prune(self.snapshot)?;
```
### To Reproduce
Related delta-rs issue: https://github.com/delta-io/delta-rs/issues/3278#issuecomment-2691264921
### Expected behavior
Allow literal value coercion during pruning
### Additional context
_No response_
Contributor guide
Research direction
Start in delta-rs/crates/core/src/delta_datafusion/mod.rs and trace PruningPredicate::try_new followed by prune using the predicate shown in the issue. Check how mismatched string literals and int columns are handled, then verify that safe literal coercion enables pruning without changing valid results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100