pingcap / pingcap/tidb

executor: push down table_id/index_id filters for TIFLASH_INDEXES system table

Open
#68,821 2 comments 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

`ADD VECTOR INDEX` progress polling against `information_schema.tiflash_indexes` should push `table_id` / `index_id` predicates down to TiFlash.

## Problem

When a cluster has many TiFlash vector-index tables, the DDL progress checker queries `information_schema.tiflash_indexes` for a specific table/index. Today `TiFlashSystemTableExtractor` only extracts predicates such as TiFlash instance, database, and table name for TiFlash system-table requests. It does not extract `table_id` / `index_id`, so the TiDB retriever sends a broad TiFlash system-table query.

At 100k vector-index table scale, this makes progress polling scale with all TiFlash local-index metadata instead of the target table/index.

Recent profiling evidence:

- Capture `2026-05-31 17:55 CST`: about 87,762 vector-index tables were ready and `ADD VECTOR INDEX` was still running.
- Capture `2026-06-01 08:33 CST`: about 99.9k vector-index tables were ready.
- TiDB CPU profile: `TiFlashSystemTableRetriever.dataForTiFlashSystemTables` was a major TiDB CPU consumer; `encoding/json.Unmarshal` and GC were also visible.
- TiDB allocation profile: `TiFlashSystemTableRetriever.dataForTiFlashSystemTables` and `LoadTiFlashReplicaInfo` allocated heavily during progress polling.
- TiFlash CPU profiles showed the companion bottleneck in `StorageSystemDTLocalIndexes::read` / `DeltaMergeStore::getLocalIndexStats`.

This issue tracks the TiDB side of a two-layer fix. TiFlash should also prune early before calling `getLocalIndexStats()`.

## Proposed change

- Extend `TiFlashSystemTableExtractor` for `information_schema.tiflash_indexes` / `TableTiFlashIndexes` to extract `table_id` and `index_id` predicates.
- Support at least equality and `IN` predicates, including conjunctions with the existing extracted predicates.
- Carry the extracted `table_id` / `index_id` filters into the TiFlash system-table request SQL so TiFlash can prune by exact table/index.
- Keep behavior unchanged for other TiFlash system tables.
- Handle contradiction / empty ranges by skipping the TiFlash request when possible.
- Add tests for equality, `IN`, conjunctions, and non-pushable predicates.

## Acceptance criteria

- `EXPLAIN` or equivalent executor diagnostics show that `table_id` / `index_id` filters are extracted for `information_schema.tiflash_indexes`.
- `ADD VECTOR INDEX` progress polling sends a targeted TiFlash system-table query containing the relevant `table_id` / `index_id` filter.
- Combined with the TiFlash-side pruning change, progress polling cost becomes proportional to the matched table/index instead of all TiFlash vector-index tables.

Contributor guide

Open the contributing guide

Research direction

Locate TiFlashSystemTableExtractor and the TableTiFlashIndexes handling, then trace how extracted predicates reach TiFlashSystemTableRetriever.dataForTiFlashSystemTables. Check existing extractor and executor diagnostics tests first; done means equality, IN, conjunction, non-pushable, and contradiction cases are covered and the generated request retains table_id/index_id filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.