apache / apache/datafusion

Should `PruningPredicate` coerce?

Open
#14,944 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Currently when you pass a pruning predicate where the predicate has a different type as the targeted column it will not prune it, even though in theory the value is castable to the target column type.

The predicate looks like this: "month_id = '202502' AND date_id = '20250226'"

However the columns are `int` column not `utf8`. So in theory these string values can be casted to int but I don't believe this is happening. Is this something that should be added

This is our rust code btw

```
// delta-rs/crates/core/src/delta_datafusion/mod.rs
let pruning_predicate = PruningPredicate::try_new(predicate.clone(), logical_schema.clone())?;
let files_to_prune = pruning_predicate.prune(self.snapshot)?;
```

### To Reproduce

Related delta-rs issue: https://github.com/delta-io/delta-rs/issues/3278#issuecomment-2691264921

### Expected behavior

Allow literal value coercion during pruning

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in delta-rs/crates/core/src/delta_datafusion/mod.rs and trace PruningPredicate::try_new followed by prune using the predicate shown in the issue. Check how mismatched string literals and int columns are handled, then verify that safe literal coercion enables pruning without changing valid results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.