pingcap / pingcap/tidb

planner: correlated equality is not considered a constant index prefix for order property matching

Open
#69,156 2 comments 0 reactions 0 assignees View on GitHub
affects-8.5 planner/performance report/customer severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Create a table with a composite index whose first column is constrained by a correlated equality and whose second column provides the required order:

```sql
CREATE TABLE accounts (
id BIGINT PRIMARY KEY,
user_id BIGINT,
KEY idx_user_id(user_id)
);

CREATE TABLE position_snapshots (
id BIGINT PRIMARY KEY,
account_id BIGINT NOT NULL,
snapshot_at DATETIME(6) NOT NULL,
payload JSON,
KEY idx_position_snapshots_snapshot_at(account_id, snapshot_at)
);

EXPLAIN
SELECT a.id,
(
SELECT ps.payload
FROM position_snapshots ps
WHERE ps.account_id = a.id
ORDER BY ps.snapshot_at DESC
LIMIT 1
) AS latest_snapshot
FROM accounts a
WHERE a.user_id = 8295;
```

In the observed plan, the inner side of `Apply` uses an unordered lookup followed by a two-stage TopN:

```text
Apply
└─TopN(root) snapshot_at:desc, count:1
└─IndexLookUp
├─IndexRangeScan account_id = CorrelatedColumn(a.id), keep order:false
└─TopN(cop) snapshot_at:desc, count:1
└─TableRowIDScan
```

The issue is reproducible even though an index on `(account_id, snapshot_at)` is available.

### 2. What did you expect to see? (Required)

For each `Apply` probe, `a.id` is a runtime constant. Therefore:

```sql
ps.account_id = a.id
```

fixes the first index column for that probe, and `(account_id, snapshot_at)` can provide `snapshot_at DESC` by scanning the matching index range in reverse order.

The optimizer should be able to enumerate an ordered inner path similar to:

```text
Apply
└─Limit 1
└─IndexLookUp
└─IndexRangeScan(account_id = CorrelatedColumn(a.id))
keep order:true, desc:true
```

The logical TopN already enumerates a `PhysicalLimit` alternative that requests the `snapshot_at DESC` property. The composite index should be able to satisfy that property because its first column is fixed by the correlated equality for each probe.

### 3. What did you see instead (Required)

The `PhysicalLimit` alternative is rejected during access-path property matching, leaving only the unordered path plus TopN.

The planner already treats `col = CorrelatedColumn` as a runtime-constant equality for range construction:

1. `SplitCorColAccessCondFromFilters` moves `index_col = correlated_col` from table filters into `AccessConds`.
2. `ResolveCorrelatedColumns` rebuilds the range using the current outer-row value during execution.

However, after the correlated condition is appended to `AccessConds`, its constant-prefix property is not propagated to the metadata used by `matchProperty`. In particular, `AccessPath.ConstCols` is populated only from `DetachRangeResult.ColumnValues`, which represents values available during static range detachment.

`matchProperty` can skip an index prefix only when either:

- `path.ConstCols[colIdx]` is true, or
- the statically built ranges prove that the index column is a point value.

A correlated equality satisfies neither check at planning time, even though it is fixed for each `Apply` probe. Consequently, `(account_id, snapshot_at)` is incorrectly considered unable to satisfy `ORDER BY snapshot_at DESC`.

Relevant code paths:

- `pkg/planner/util/path.go`: `SplitCorColAccessCondFromFilters`
- `pkg/planner/core/stats.go`: `deriveIndexPathStats` and `detachCondAndBuildRangeForPath`
- `pkg/planner/core/find_best_task.go`: `matchProperty`

The key issue is that the runtime-constant nature of the correlated access condition is used for range construction but not for order-property matching.

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.6
Git Commit Hash: ae18096e023780bb56bfce33698abec0d4640d0a
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.