pingcap / pingcap/tidb

planner: optimizer range building memory is not tracked for long IN lists

Open
#67,817 0 comments 0 reactions 0 assignees View on GitHub
report/customer severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This is an observability and memory-attribution issue in the planner.

A minimal scenario:

1. Enable OOM/top SQL memory diagnostics, or inspect SQL/session memory through the existing memory tracker based diagnostics.
2. Run a statement with a very long `IN` list on an indexed column, for example:

```sql
CREATE TABLE t (a VARCHAR(64), b INT, KEY idx_a(a));
EXPLAIN SELECT * FROM t WHERE a IN (... tens or hundreds of thousands of constants ...);
```

3. Observe memory growth while the optimizer builds ranges for the long `IN` list.

Code evidence from current master:

- `RangerContext` does not carry a query/session `MemTracker`: `pkg/util/ranger/context/context.go`.
- `builder.buildFromIn` allocates range points proportional to `len(list) * 2`, copies datum values, then sorts/deduplicates them without calling `MemTracker.Consume`: `pkg/util/ranger/points.go`.
- `points2Ranges` estimates memory only for `tidb_opt_range_max_size` fallback; it does not account the allocation to the SQL memory tracker: `pkg/util/ranger/ranger.go`.
- OOM/top SQL memory attribution depends on `Tracker.Consume` and `ProcessInfo.MemTracker.MaxConsumed()`: `pkg/util/memory/tracker.go`, `pkg/util/memoryusagealarm/memoryusagealarm.go`.
- The cached-plan range rebuild path calls ranger with `rangeMaxSize = 0`, so the normal range-size fallback is not applied there: `pkg/planner/core/plan_cache_rebuild.go`.

### 2. What did you expect to see? (Required)

Transient memory allocated while building optimizer ranges for long `IN` lists should be attributed to the current SQL/session memory tracker, or otherwise be visible in OOM/top SQL diagnostics.

`tidb_opt_range_max_size` fallback and plan-cache range rebuilding should not leave an observability blind spot where the memory is consumed by the current SQL but cannot be traced back to it by TiDB diagnostics.

### 3. What did you see instead (Required)

The memory used during optimizer range-point/range construction is not consistently recorded by the SQL/session `MemTracker`.

As a result, when a long `IN` list causes high transient memory usage during optimization, OOM alarm/top SQL diagnostics can miss or underreport the SQL responsible. `tidb_opt_range_max_size` may prevent some range explosion cases by falling back, but that is separate from memory attribution, and the cached-plan range rebuild path currently bypasses that limit by passing `rangeMaxSize = 0`.

### 4. What is your TiDB version? (Required)

Observed from current master code inspection on 2026-04-16. Exact affected released versions have not been verified.

### Additional context

Related but not a duplicate: #37176 tracks the broader memory-control/refactor problem for building ranges. This issue is specifically about memory tracking and OOM/top SQL attribution for optimizer range building with long `IN` lists.

Suggested labels: `type/bug`, `sig/planner`, `component/planner`, `severity/moderate`, `may-affects-master`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.