sys: improve the memory usage of scanning INFORMATION_SCHEMA.TIDB_INDEX_USAGE
Open
epic/memory-management
sig/execution
sig/planner
sig/sql-infra
type/enhancement
type/performance
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
`INFORMATION_SCHEMA.TIDB_INDEX_USAGE` is used to help users to manage their indexes, for example, remove unused indexes to improve system performance.
But this table is a memory table, and when scanning it, we don't have any optimization, which means we have to scan all data, convert it into rows in memory, and filter unnecessary rows.
The whole process could cost lots of memory for SaaS users who have millions of tables, and finally cause an OOM error.
Below is an example, this user has around 3000,00 tables and 3000,000 indexes, the memory quota of a query is 1G by default:
Contributor guide
Assessment
This issue has not been assessed yet.