apache / apache/datafusion

Unified `TableScan.filters`

Open
#19,894 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Problem Statement

The current DML filter extraction implementation in DataFusion exhibits a **architectural inconsistency**: the `TableScan` logical plan node maintains filters in two distinct ways:

1. **Filters passed to TableProvider** - Via `delete_from()`, `update()`, etc., operators
2. **Filters in LogicalPlan** - Via `TableScan.filters` field during planning

This dual-track approach creates several problems:

### Current Issues

1. **Filter Duplication Risk**
- Same predicate may exist in both `Filter` node and `TableScan.filters`
- Deduplication logic becomes complex and error-prone
- Different code paths may process the same filter inconsistently

2. **Semantic Confusion**
- Unclear which filters are "pushed down" vs. "logical"
- Makes it difficult to reason about query semantics
- Complicates optimizer correctness proofs

3. **Implementation Burden**
- DML operations must collect filters from multiple locations
- Qualifier-stripping and validation needed at collection time
- Each new operator/optimizer rule must handle both locations

4. **Multi-Table Safety Hazards**
- UPDATE...FROM scenarios require careful tracking of which table each filter belongs to
- Target table scoping becomes a bandage rather than systemic solution
- Cross-table predicate contamination possible if new patterns emerge

### Example Scenario (UPDATE...FROM)

```sql
UPDATE target SET col = val
FROM source
WHERE target.id = source.id AND target.status = 'active'
```

**Current fragility:**
- Filters could appear in: Filter node, target TableScan.filters, or source TableScan.filters
- DML must track: which filters belong to target vs. source
- Risk: Source filter accidentally applied to target during extraction

**With unified design:**
- Single, clear filter representation
- Optimizer ensures filters on each table are stored consistently
- DML extraction becomes straightforward and safe

---

## Proposed Solution: Unified Filter Field

### Design Goals

1. **Single Source of Truth** - One representation of filters for each table
2. **Explicit Semantics** - Clear distinction between logical and pushed-down filters
3. **Safety** - Impossible for filters to be duplicated or cross-contaminated
4. **Performance** - No additional overhead vs. current approach

### Approach

Introduce a **unified filter contract** where:

1. **TableScan Becomes the Filter Container**
- `TableScan.filters` becomes the *only* place where table predicates live during planning
- Filter nodes are removed/consolidated during logical planning (not pushed down later)

2. **Filter Node Redesign**
- Retains Filter node for predicates that can't be expressed in TableScan
- Examples: complex expressions, cross-table joins, subqueries
- Simple table-local predicates live in TableScan.filters

3. **Optimizer Clarity**
- "Push down" means: move predicate from Filter into TableScan.filters
- "Pull up" means: move predicate from TableScan.filters into Filter (rare)
- Symmetry and clarity in optimizer rules

### Benefits

| Benefit | Current | Unified | Impact |
|---------|---------|---------|--------|
| Filter extraction | Multiple locations | Single location | Simpler DML logic |
| Cross-table safety | Complex tracking | Scoped by design | Safer UPDATE...FROM |
| Deduplication | Needed at runtime | Impossible structurally | Fewer bugs |
| Optimizer rules | Must handle both | Single representation | Cleaner code |
| Reasoning about plans | Unclear semantics | Explicit semantics | Better debugging |

---

@adriangb 's [suggestion in #19884](https://github.com/apache/datafusion/pull/19884#issuecomment-3769536637)

Contributor guide

Open the contributing guide

Research direction

Review TableScan and the delete_from()/update() planning paths first, using the UPDATE...FROM example to trace where predicates currently reside. Done means establishing one scoped TableScan.filters contract, retaining Filter nodes only for non-table-local predicates, and removing multi-location handling from DML extraction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
database
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.