pingcap / pingcap/tidb

I create a cluster with 2M users, and use BR to backup / restore it, meet error

Open
#59,673 1 comment 0 reactions 0 assignees View on GitHub
component/br may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

I create a cluster with 2M users:

```
mysql> use mysql;
Database changed

mysql> select count(*) from user;
+----------+
| count(*) |
+----------+
| 2494895 |
+----------+
1 row in set (0.71 sec)
```

Use br to backup:

```
tiup br:nightly backup full --storage "local:///tmp/backup"
```

and restore it:

```
tiup playground nightly --host 0.0.0.0 --tiflash 0 --db.binpath ./bin/tidb-server
tiup br:nightly restore full --storage "local:///tmp/backup"
```

### 2. What did you expect to see? (Required)

Success

### 3. What did you see instead (Required)

```
tiup br:nightly restore full --storage "local:///tmp/backup"
Starting component br: /home/genius/.tiup/components/br/v9.0.0-alpha-nightly/br restore full --storage local:///tmp/backup
Detail BR log in /tmp/br.log.2025-02-20T15.21.56+0800
Split&Scatter Regions <---------------------------------------------------------------------------------------> 100.00%
Download&Ingest SST <-----------------------------------------------------------------------------------------> 100.00%
Restore Pipeline <--------------------------------------------------------------------------------------------> 100.00%
[2025/02/20 15:22:19.354 +08:00] [INFO] [collector.go:77] ["Full Restore failed summary"] [total-ranges=240] [ranges-succeed=240] [ranges-failed=0] [merge-ranges=798.965µs] [split-regions=125.662394ms] [restore-files=2.746380622s] [restore-pipeline=5.142510093s] [default-CF-files=47] [write-CF-files=193] [split-keys=9]
Error: error during merging temporary tables into system tables, table: global_priv: [BR:Common:ErrUnknown]failed to execute REPLACE INTO mysql.global_priv(host,user,priv) SELECT host,user,priv FROM __tidb_br_temporary_mysql.global_priv;: [kv:8004]Transaction is too large, size: 104857618
Error: run /home/genius/.tiup/components/br/v9.0.0-alpha-nightly/br (wd:/home/genius/.tiup/data/UdKIDAX) failed: exit status 1
```

It use a single `REPLACE INTO ` for the mysql tables, when there are 2M users, the transaction is too large.

### 4. What is your TiDB version? (Required)

master 2025-02-20

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the documented tiup br:nightly backup and restore commands with a large mysql.user table, then trace the restore step that merges into mysql.global_priv and the temporary table. Done means the full restore completes successfully without the 104857618-byte transaction-too-large error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.