pingcap / pingcap/tidb

br node OOM when running restore cmd manullay inside a pod with 56GB memory

Open
#50,631 1 comment 0 reactions 0 assignees View on GitHub
component/br severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1. deploy a br pod with below configuration in the GCP n2-standard-16 host.
```
requests:
cpu: 15000m
memory: 56Gi
limits:
cpu: 16000m
memory: 64Gi
```
2. connect to the pod via shell, and run br restore command to restore a br backup (no stats in the backup) of 300TB cluster to a new TiDB TiDB cluster with 100 TiKV node

### 2. What did you expect to see? (Required)
br node should not OOM.

### 3. What did you see instead (Required)
br node OOM.
When repeat the testing with 128gb br node, br node doesn't OOM.

### 4. What is your TiDB version? (Required)
v7.6.0

Contributor guide

Open the contributing guide

Research direction

Start with the BR restore command in the reported TiDB v7.6.0 environment and reproduce it in a pod with 56Gi memory while restoring the 300TB backup to a 100-TiKV cluster. Investigate the cause of the BR node OOM and define the resource behavior expected under that configuration; done means the restore no longer OOMs with 56Gi memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.