pingcap / pingcap/tidb

Account for memory used by INFORMATION_SCHEMA metadata queries

Open
#70,431 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

### Problem

Memory allocated by `INFORMATION_SCHEMA` memory tables, especially `information_schema.tables`, is not fully accounted for by TiDB's statement memory tracker.

A metadata query may need to enumerate a large number of table definitions and materialize metadata rows and derived values. However, the actual process-level memory usage can grow substantially while statement-level memory metrics such as `MEMORY_CURRENT` and slow-log `Mem_max` report a much smaller value.

This causes the following problems:

- Excessive-memory metadata queries are difficult to identify.
- Statement-level memory metrics do not accurately reflect the query's memory consumption.
- Query memory quota enforcement may not cover all memory allocated by the query.
- Untracked memory may grow until it contributes to a process-level OOM.

### Expected behavior

Memory allocated on behalf of an `INFORMATION_SCHEMA` metadata query should be accounted for by the corresponding statement memory tracker.

The reported statement memory, slow-log memory fields, and query memory quota enforcement should reasonably reflect the actual memory consumed by the query. If the query exceeds its memory quota, TiDB should terminate it with a controlled memory-quota error instead of allowing untracked allocations to cause a process-level OOM.

### Proposed work

1. Make memory allocated by `INFORMATION_SCHEMA` metadata queries visible to the statement memory tracker and query memory quota enforcement.
2. Add a regression test with a large metadata catalog to verify that the reported query memory is consistent with the actual memory usage.

Contributor guide

Open the contributing guide

Research direction

Start by tracing INFORMATION_SCHEMA memory tables, especially information_schema.tables, into the statement memory tracker and query memory quota enforcement. Use a large metadata catalog for the regression test and compare process usage with MEMORY_CURRENT and slow-log Mem_max. Done means allocations are tracked and quota excess produces a controlled memory-quota error rather than a process-level OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.