storage: prune dt_local_indexes by table_id/index_id before getLocalIndexStats
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
StorageSystemDTLocalIndexes::read should prune by table_id / index_id before calling getLocalIndexStats().
Problem
system.dt_local_indexes currently parses keyspace_id from the query, then iterates DeltaMerge tables and calls getLocalIndexesStatsFromStorage(dm_storage) before applying any table_id / index_id pruning.
At 100k vector-index table scale, TiDB DDL progress polling for ADD VECTOR INDEX repeatedly queries TiFlash local-index status for one target table/index. Without early pruning, TiFlash scans local-index metadata across many tables for every poll.
Recent profiling evidence:
- Capture
2026-05-31 17:55 CST: about 87,762 vector-index tables were ready andADD VECTOR INDEXwas still running. - Capture
2026-06-01 08:33 CST: about 99.9k vector-index tables were ready. - TiFlash-1 CPU profile over 30s:
StorageSystemDTLocalIndexes::readaccounted for about 96.01% cumulative CPU underGetTiFlashSystemTable;DeltaMergeStore::getLocalIndexStatsaccounted for about 35.97%. - TiFlash-2 CPU profile over 30s:
StorageSystemDTLocalIndexes::readaccounted for about 96.60% cumulative CPU;DeltaMergeStore::getLocalIndexStatsaccounted for about 37.25%. - Related work inside metadata expansion included
DMFileMetaV2::getLocalIndexStateandgenerateLocalIndexInfos.
This issue tracks the TiFlash side of a two-layer fix. TiDB should also push table_id / index_id predicates down when querying information_schema.tiflash_indexes.
Proposed change
- Extend system-table predicate parsing for
system.dt_local_indexesto recognizetable_idandindex_id, at least for equality andINpredicates combined byAND. - In
StorageSystemDTLocalIndexes::read, skip nonmatchingtable_info.idbefore callinggetLocalIndexesStatsFromStorage(dm_storage)/getLocalIndexStats(). - Apply
index_idfiltering before expanding per-index stats where possible. - If current
getLocalIndexStats()APIs cannot avoid scanning all local indexes / DMFiles for a table, add a filtered variant or pass a filter into the helper. - Preserve existing behavior when no
table_id/index_idfilter is present.
Acceptance criteria
SELECT * FROM system.dt_local_indexes WHERE table_id = X AND index_id = Yavoids scanning other tables and avoids callinggetLocalIndexStats()for nonmatching tables.- Predicate pruning is covered by unit or integration tests for equality,
IN, conjunctions, and no-filter fallback. - With the TiDB-side pushdown change,
ADD VECTOR INDEXprogress polling at 100k table scale is proportional to the matched table/index rather than all local-index tables.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in StorageSystemDTLocalIndexes::read and trace the system-table predicate parsing, then inspect getLocalIndexesStatsFromStorage and getLocalIndexStats. Implement pruning for table_id and index_id, preserving the no-filter path, and add unit or integration coverage for equality, IN, conjunctions, and fallback behavior. Verify that matching queries avoid scanning unrelated tables and indexes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100