NUMA-Aware Memory Control Failure: Global Memory Controller Ignores NUMA Node Constraints, Causing TiDB OOM
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
When using tiup to bind TiDB to a specific NUMA node, the global memory controller fails to recognize NUMA node-local memory limits. Instead, it incorrectly calculates thresholds based on total system memory, resulting in:
- Overestimated memory thresholds
- TiDB process exceeding actual NUMA node capacity
- Frequent OOM kills by the system
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. Deploy TiDB via TiUP with NUMA binding:
```yaml
tidb_servers:
- host: 172.16.6.xx
ssh_port: 22
port: 4000
status_port: 10080
deploy_dir: /data2/perftest/deploy/tidb-4000
log_dir: /data2/perftest/deploy/tidb-4000/log
numa_node: "1"
arch: amd64
os: linux
```
2. check the memory settings
```shell
> numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46
node 0 size: 94763 MB
node 0 free: 77131 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
node 1 size: 96758 MB
node 1 free: 92394 MB
node distances:
node 0 1
0: 10 21
1: 21 10
```
```
mysql> select * from INFORMATION_SCHEMA.MEMORY_USAGE;
+--------------+--------------+----------------+-----------------+-------------+-------------------+--------------------+---------------------+----------+------------+------------------+
| MEMORY_TOTAL | MEMORY_LIMIT | MEMORY_CURRENT | MEMORY_MAX_USED | CURRENT_OPS | SESSION_KILL_LAST | SESSION_KILL_TOTAL | GC_LAST | GC_TOTAL | DISK_USAGE | QUERY_FORCE_DISK |
+--------------+--------------+----------------+-----------------+-------------+-------------------+--------------------+---------------------+----------+------------+------------------+
| 200825729024 | 160660583200 | 252084224 | 399417344 | NULL | NULL | 0 | 0001-01-01 00:00:00 | 0 | 0 | 0 |
+--------------+--------------+----------------+-----------------+-------------+-------------------+--------------------+---------------------+----------+------------+------------------+
1 row in set (0.00 sec)
```
3. Generate memory pressure exceeding the bound NUMA node's capacity but below total system memory
```
set @@cte_max_recursion_depth=100000000;
set @@tidb_tmp_table_max_size=137438953472;
set tidb_dml_type = "bulk";
set tidb_mem_quota_query=0;
CREATE TABLE IF NOT EXISTS large_data (
id BIGINT PRIMARY KEY,
random_str VARCHAR(255),
num1 DOUBLE,
num2 DOUBLE,
json_data JSON,
index idx_num1 (num1),
index idx_random (random_str(10))
);
-- insert 1 million rows
INSERT INTO large_data
WITH RECURSIVE cte AS (
SELECT 1 AS n,
CONCAT('str_', UUID()) AS rand_str,
RAND() * 1000 AS n1,
RAND() * 10000 AS n2,
JSON_OBJECT('key', SHA(RAND())) AS j
UNION ALL
SELECT n + 1,
CONCAT('str_', UUID()),
RAND() * 1000,
RAND() * 10000,
JSON_OBJECT('key', SHA(RAND()))
FROM cte
WHERE n < 1000000
)
SELECT * FROM cte;
SELECT
t1.id,
t2.id,
t3.id,
CONCAT(
SUBSTRING(t1.random_str, 1, 10),
SUBSTRING(t2.random_str, 5, 15),
SUBSTRING(t3.random_str, 10, 20)
) AS combined_string,
SQRT(t1.num1 * t2.num2 * t3.num1 * t3.num2) * EXP(t1.num1 / 100) AS complex_calculation,
AVG(t1.num1) OVER (PARTITION BY t1.id % 1000 ORDER BY t1.num2) AS moving_avg,
JSON_MERGE_PATCH(
t1.json_data,
JSON_OBJECT('combined', JSON_ARRAY(t2.num1, t3.num2))
) AS merged_json,
(SELECT COUNT(*) FROM large_data WHERE num1 BETWEEN t1.num1 AND t1.num1 + 100) AS range_count,
SUM(t2.num2) OVER (ORDER BY t1.num1 ROWS BETWEEN 100 PRECEDING AND CURRENT ROW) AS sliding_sum
FROM
large_data t1
JOIN
large_data t2 ON t1.id % 1000 = t2.id % 1000 AND t1.num1 < t2.num1
JOIN
large_data t3 ON t2.id % 500 = t3.id % 500 AND t2.num2 > t3.num2
WHERE
t1.num1 > 100 AND t2.num2 < 9000
ORDER BY
complex_calculation DESC,
LENGTH(combined_string)
LIMIT 100000;
```
### 2. What did you expect to see? (Required)
TiDB is up and the big query would be killed by the global memory controller
### 3. What did you see instead (Required)
- tidb-server memory usage in Grafana shows system-wide usage

- TiDB process got killed by OOM killer, the oom_record was empty and 'global memory controller' was not shown in tidb log.

- set `tidb_server_memory_limit` to less than 93GB (the memory of one numa node) and rerun the big query. TiDB was not killed and got the following error.
```
ERROR 8176 (HY000): Your query has been cancelled due to exceeding the allowed memory limit for the tidb-server instance and this query is currently using the most memory. Please try narrowing your query scope or increase the tidb_server_memory_limit and try again.[conn=1205862406]
```
### 4. What is your TiDB version? (Required)
v8.5.1
Contributor guide
Assessment
This issue has not been assessed yet.