pingcap / pingcap/tidb

br: default backup parameter table-concurrency is too high

Open
#59,376 0 comments 0 reactions 0 assignees View on GitHub
component/br type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Run BR backup

### 2. What did you expect to see? (Required)

We expect BR backup should use some cpu/io resource, but not too high to impact cluster fuction

### 3. What did you see instead (Required)

There is a spike at the beginning of checksum stage. It caused a production outage of a paid customer. Further analysis find that default value for table-concurrency is 64, and default value for checksum-concurrency is 4. This caused a total concurrency of 256 of intensive cpu/io utilization.

Since 8.5 BR backup checksum is disabled by default. However there is still lack of documentation on table-concurrency and checksum-concurrency, users may risk cluster outage when performing BR backup with checksum enabled.

The default value of table-concurrency should be reduced to a level that it is safe to run without adjusting it, but allow user to increase if they want better performance.

### 4. What is your TiDB version? (Required)

select tidb_version()\G
*************************** 1. row ***************************
tidb_version(): Release Version: v7.5.5
Edition: Community
Git Commit Hash: 287f27ea0e5b7be6d75bd4c6fc9982f9d80e4cf7
Git Branch: HEAD
UTC Build Time: 2024-12-26 07:05:21
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.