Unified `TableScan.filters`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Problem Statement
The current DML filter extraction implementation in DataFusion exhibits a **architectural inconsistency**: the `TableScan` logical plan node maintains filters in two distinct ways:
1. **Filters passed to TableProvider** - Via `delete_from()`, `update()`, etc., operators
2. **Filters in LogicalPlan** - Via `TableScan.filters` field during planning
This dual-track approach creates several problems:
### Current Issues
1. **Filter Duplication Risk**
- Same predicate may exist in both `Filter` node and `TableScan.filters`
- Deduplication logic becomes complex and error-prone
- Different code paths may process the same filter inconsistently
2. **Semantic Confusion**
- Unclear which filters are "pushed down" vs. "logical"
- Makes it difficult to reason about query semantics
- Complicates optimizer correctness proofs
3. **Implementation Burden**
- DML operations must collect filters from multiple locations
- Qualifier-stripping and validation needed at collection time
- Each new operator/optimizer rule must handle both locations
4. **Multi-Table Safety Hazards**
- UPDATE...FROM scenarios require careful tracking of which table each filter belongs to
- Target table scoping becomes a bandage rather than systemic solution
- Cross-table predicate contamination possible if new patterns emerge
### Example Scenario (UPDATE...FROM)
```sql
UPDATE target SET col = val
FROM source
WHERE target.id = source.id AND target.status = 'active'
```
**Current fragility:**
- Filters could appear in: Filter node, target TableScan.filters, or source TableScan.filters
- DML must track: which filters belong to target vs. source
- Risk: Source filter accidentally applied to target during extraction
**With unified design:**
- Single, clear filter representation
- Optimizer ensures filters on each table are stored consistently
- DML extraction becomes straightforward and safe
---
## Proposed Solution: Unified Filter Field
### Design Goals
1. **Single Source of Truth** - One representation of filters for each table
2. **Explicit Semantics** - Clear distinction between logical and pushed-down filters
3. **Safety** - Impossible for filters to be duplicated or cross-contaminated
4. **Performance** - No additional overhead vs. current approach
### Approach
Introduce a **unified filter contract** where:
1. **TableScan Becomes the Filter Container**
- `TableScan.filters` becomes the *only* place where table predicates live during planning
- Filter nodes are removed/consolidated during logical planning (not pushed down later)
2. **Filter Node Redesign**
- Retains Filter node for predicates that can't be expressed in TableScan
- Examples: complex expressions, cross-table joins, subqueries
- Simple table-local predicates live in TableScan.filters
3. **Optimizer Clarity**
- "Push down" means: move predicate from Filter into TableScan.filters
- "Pull up" means: move predicate from TableScan.filters into Filter (rare)
- Symmetry and clarity in optimizer rules
### Benefits
| Benefit | Current | Unified | Impact |
|---------|---------|---------|--------|
| Filter extraction | Multiple locations | Single location | Simpler DML logic |
| Cross-table safety | Complex tracking | Scoped by design | Safer UPDATE...FROM |
| Deduplication | Needed at runtime | Impossible structurally | Fewer bugs |
| Optimizer rules | Must handle both | Single representation | Cleaner code |
| Reasoning about plans | Unclear semantics | Explicit semantics | Better debugging |
---
@adriangb 's [suggestion in #19884](https://github.com/apache/datafusion/pull/19884#issuecomment-3769536637)
Contributor guide
Research direction
Review TableScan and the delete_from()/update() planning paths first, using the UPDATE...FROM example to trace where predicates currently reside. Done means establishing one scoped TableScan.filters contract, retaining Filter nodes only for non-table-local predicates, and removing multi-location handling from DML extraction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- database
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100