Account for memory used by INFORMATION_SCHEMA metadata queries
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
### Problem
Memory allocated by `INFORMATION_SCHEMA` memory tables, especially `information_schema.tables`, is not fully accounted for by TiDB's statement memory tracker.
A metadata query may need to enumerate a large number of table definitions and materialize metadata rows and derived values. However, the actual process-level memory usage can grow substantially while statement-level memory metrics such as `MEMORY_CURRENT` and slow-log `Mem_max` report a much smaller value.
This causes the following problems:
- Excessive-memory metadata queries are difficult to identify.
- Statement-level memory metrics do not accurately reflect the query's memory consumption.
- Query memory quota enforcement may not cover all memory allocated by the query.
- Untracked memory may grow until it contributes to a process-level OOM.
### Expected behavior
Memory allocated on behalf of an `INFORMATION_SCHEMA` metadata query should be accounted for by the corresponding statement memory tracker.
The reported statement memory, slow-log memory fields, and query memory quota enforcement should reasonably reflect the actual memory consumed by the query. If the query exceeds its memory quota, TiDB should terminate it with a controlled memory-quota error instead of allowing untracked allocations to cause a process-level OOM.
### Proposed work
1. Make memory allocated by `INFORMATION_SCHEMA` metadata queries visible to the statement memory tracker and query memory quota enforcement.
2. Add a regression test with a large metadata catalog to verify that the reported query memory is consistent with the actual memory usage.
Contributor guide
Research direction
Start by tracing INFORMATION_SCHEMA memory tables, especially information_schema.tables, into the statement memory tracker and query memory quota enforcement. Use a large metadata catalog for the regression test and compare process usage with MEMORY_CURRENT and slow-log Mem_max. Done means allocations are tracked and quota excess produces a controlled memory-quota error rather than a process-level OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100