Support multi-stage late materialization for DeltaMerge TableScan
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
Support multi-stage late materialization for TiFlash DeltaMerge TableScan when a query has both pushed-down TableScan filters and residual Selection filters.
Tracking implementation PR: #10999
Motivation
For wide table scans, the current read path can still read many output columns for rows that will later be removed by residual filters. This is inefficient when the residual filter only depends on a small subset of columns and has good selectivity.
Multi-stage late materialization should reduce unnecessary column reads by splitting scan work into multiple stages:
- Stage 0 reads columns needed by the pushed-down TableScan filter and applies rough-set / bitmap filtering.
- Stage 1 reads only residual-filter columns for candidate rows and evaluates the residual filter.
- The final stage reads the remaining output columns only for rows that pass previous stages.
Proposal
Add a guarded DeltaMerge read path that can split TableScan columns into stage0 filter columns, stage1 residual-filter columns, and final remaining columns. The implementation should preserve existing query semantics, including extra casts after TableScan, virtual-column handling, generated-column restrictions, remote read behavior, and runtime execution summaries.
Acceptance Criteria
- The optimization is controlled by
dt_enable_multi_stage_late_materialization. - The optimization is only enabled for supported paths and is disabled for unsafe cases such as generated columns, keep-order scans, fast scan, force-pushdown-all-filters, and stage1 filters referencing virtual
_tidb_tid. - Query results match the existing non-MSLM path.
- TableScan and Selection
actRowsremain correct for local, remote-only, and mixed local/remote execution. - Unit tests cover stage splitting, residual filter evaluation, direct/late final reads, extra casts, hidden commit_ts / virtual column edge cases, and runtime row reporting.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with implementation PR #10999 and the existing DeltaMerge TableScan read path. Trace how pushed-down and residual filters are handled, then use the listed unit-test cases to verify stage splitting, query results, and runtime row reporting across local and remote execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100