pingcap / pingcap/tiflash

Support multi-stage late materialization for DeltaMerge TableScan

Open
#11,003 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-8.5 type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

Support multi-stage late materialization for TiFlash DeltaMerge TableScan when a query has both pushed-down TableScan filters and residual Selection filters.

Tracking implementation PR: #10999

Motivation

For wide table scans, the current read path can still read many output columns for rows that will later be removed by residual filters. This is inefficient when the residual filter only depends on a small subset of columns and has good selectivity.

Multi-stage late materialization should reduce unnecessary column reads by splitting scan work into multiple stages:

  • Stage 0 reads columns needed by the pushed-down TableScan filter and applies rough-set / bitmap filtering.
  • Stage 1 reads only residual-filter columns for candidate rows and evaluates the residual filter.
  • The final stage reads the remaining output columns only for rows that pass previous stages.

Proposal

Add a guarded DeltaMerge read path that can split TableScan columns into stage0 filter columns, stage1 residual-filter columns, and final remaining columns. The implementation should preserve existing query semantics, including extra casts after TableScan, virtual-column handling, generated-column restrictions, remote read behavior, and runtime execution summaries.

Acceptance Criteria

  • The optimization is controlled by dt_enable_multi_stage_late_materialization.
  • The optimization is only enabled for supported paths and is disabled for unsafe cases such as generated columns, keep-order scans, fast scan, force-pushdown-all-filters, and stage1 filters referencing virtual _tidb_tid.
  • Query results match the existing non-MSLM path.
  • TableScan and Selection actRows remain correct for local, remote-only, and mixed local/remote execution.
  • Unit tests cover stage splitting, residual filter evaluation, direct/late final reads, extra casts, hidden commit_ts / virtual column edge cases, and runtime row reporting.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with implementation PR #10999 and the existing DeltaMerge TableScan read path. Trace how pushed-down and residual filters are handled, then use the listed unit-test cases to verify stage splitting, query results, and runtime row reporting across local and remote execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.