pingcap / pingcap/tidb

planner: support reference-specific predicate pushdown for shared materialized CTEs

Open
#67,762 0 comments 0 reactions 0 assignees View on GitHub
component/executor type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

TiDB already pushes some predicates from CTE consumers into the producer side, but the current behavior is still conservative for shared materialized CTEs: predicates from different references are merged into a broader producer-side filter, while the exact reference-specific predicates remain above each `CTEFullScan`.

This leaves an optimization gap for queries where each CTE reference has a narrow and different filter.

### Reproduction

The case already exists in `planreplayertest`:

- Query file: `/Users/weizhenwang/devel/opensource/planreplayertest/t/tpcds/without_tiflash/query_75_78.test`
- Result file: `/Users/weizhenwang/devel/opensource/planreplayertest/r/tpcds/without_tiflash/query_75_78.result`

Relevant query shape:

```sql
WITH all_sales AS (
SELECT d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id,
SUM(sales_cnt) AS sales_cnt, SUM(sales_amt) AS sales_amt
FROM (...)
GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id
)
SELECT ...
FROM all_sales curr_yr, all_sales prev_yr
WHERE curr_yr.i_brand_id = prev_yr.i_brand_id
AND curr_yr.i_class_id = prev_yr.i_class_id
AND curr_yr.i_category_id = prev_yr.i_category_id
AND curr_yr.i_manufact_id = prev_yr.i_manufact_id
AND curr_yr.d_year = 2002
AND prev_yr.d_year = 2001
AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2)) / CAST(prev_yr.sales_cnt AS DECIMAL(17,2)) < 0.9;
```

### Current plan shape

The consumer-specific predicates are still kept above the CTE scans:

```text
Selection(Build) eq(Column#353, 2001)
└─CTEFullScan CTE:all_sales AS prev_yr

Selection(Probe) eq(Column#346, 2002)
└─CTEFullScan CTE:all_sales AS curr_yr
```

At the same time, the producer side only gets the merged predicate:

```text
Selection or(eq(tpcds50.date_dim.d_year, 2002), eq(tpcds50.date_dim.d_year, 2001))
└─TableFullScan table:date_dim
```

So the optimizer does some CTE predicate pushdown today, but only at the level of a combined producer-side filter. It still materializes rows for both years into the shared CTE result and relies on per-reference `Selection -> CTEFullScan` afterward.

### Why this is worth enhancing

For shared materialized CTEs, there are cases where each consumer has a precise predicate on the CTE output, and the current merged pushdown can still materialize a significantly larger intermediate result than necessary.

In the example above:

- `curr_yr` only needs `d_year = 2002`
- `prev_yr` only needs `d_year = 2001`
- The current plan still builds one shared CTE result for `d_year IN (2001, 2002)`

That is correct, but not optimal.

### Expected direction

It would be useful to support a more aggressive optimization for shared non-recursive CTEs, for example:

1. Cost-based duplication/specialization of the CTE producer when different references have selective predicates.
2. Reference-specific predicate pushdown on the CTE read side when it can be proven safe.
3. Any equivalent rule that avoids materializing a broader shared CTE result than needed.

### Expected behavior

When a shared CTE is referenced multiple times and each reference has a selective predicate on CTE output columns, TiDB should be able to choose a plan that keeps those predicates closer to the producer instead of materializing an unnecessarily broad shared CTE result.

### Environment

- Source: local `planreplayertest` corpus
- Query: TPC-DS query 75 variant
- TiDB branch: local `pingcap/tidb` checkout on 2026-04-14

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.