pingcap / pingcap/tidb

infoschema v2: optimize partition table memory usage

Open
#68,494 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Problem Description

Partition tables in InfoSchema V2 are currently classified as "special attributes" and their full `PartitionInfo` is stored in `tableInfoResident`, bypassing the Sieve cache. This violates the V2 design principle of lazy loading and causes significant memory bloat for large-scale partitioned table clusters.

### Current State

For a cluster with:
- 1,000 partitioned tables
- Average 100 partitions per table

**Estimated memory waste**: ~100MB of permanent memory for `PartitionInfo` (including partition definitions, expressions, etc.)

### Root Cause

The original design assumed:
1. Background tasks need to iterate over tables with special attributes
2. Different background tasks iterate over different attribute types
3. Without resident storage, repeated iterations would be expensive

**However, this assumption is incorrect for partition tables:**
- Partition tables have the lowest access frequency among all special attributes
- No periodic background tasks iterate over partition tables
- Access patterns are primarily: startup, user-initiated queries, and DDL events

### Comparison with Other Special Attributes

| Special Attribute | Access Frequency | Background Tasks | Iteration Frequency |
|------------------|------------------|------------------|---------------------|
| TTL | ⚡⚡⚡⚡ Very High | `ttlworker` (every 5s or schema change) | Every 5s |
| TiFlash | ⚡⚡⚡ High | Async replica progress monitoring | Continuous |
| Placement | ⚡⚡ Medium | Policy validation, placement tracking | On demand |
| **Partition** | **⚡⚡ Low** | **No periodic tasks** | **Startup / Query / DDL only** |
| TableLock | ⚡ Low | Lock timeout cleanup | On demand |

### Expected Benefits

- **Memory**: Tens to hundreds of MB saved for large-scale partitioned table clusters
- **Startup**: ~95% reduction in LIST partition scan scope
- **Query Performance**: No regression (partition pruning remains O(1))

### Proposed Solution

1. Remove `PartitionAttribute` from `HasSpecialAttributes()` filter
2. Add version-aware `GetPartitionedTableIDsV2()` API (extract from `pid2tid` btree)
3. Update all callers with V2 API + V1 fallback
4. Optimize LIST partition startup rebuild (scan only LIST partitions, not all partitioned tables)

See design document for detailed analysis: `explorer/partition-table-optimization-design.md`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.