Backup(BR) is getting failed for huge table count
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Issue Description:
Our application is a SAAS based Multi-tenancy application with each tenant will have a DB , in which we have more than 14k databases and having more than 600k Tables.
While we have a strick backup requirement when we run the BR full backup its not showing any progress and gets failed with some RPC error
Command and Log:
```$ tiup br:v8.1.1 backup full --pd "10.1.3.190:2379" --storage "s3://us-chat-db-tidb-backup/test_backupdec_232024" --log-file backupdec232024.log
Starting component br: /home/ec2-user/.tiup/components/br/v8.1.1/br backup full --pd 10.1.3.190:2379 --storage s3://us-chat-db-tidb-backup/test_backupdec_232024 --log-file backupdec232024.log
Detail BR log in backupdec232024.log
Full Backup <..................................................................................................................................................................> 0.00%```
Am also attaching the details logs as well
### 2. What did you expect to see? (Required)
We expect the backup to run completely and consistently
### 3. What did you see instead (Required)
we see backup failures and BR backup is not happening
### 4. What is your TiDB version? (Required)
```mysql> select tidb_version();
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version() |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.1.1
Edition: Community
Git Commit Hash: a7df4f9845d5d6a590c5d45dad0dcc9f21aa8765
Git Branch: HEAD
UTC Build Time: 2024-08-22 05:49:03
GoVersion: go1.21.13
Race Enabled: false
Check Table Before Drop: false
Store: tikv |```[backup_log.txt](https://github.com/user-attachments/files/18239868/backup_log.txt)
Contributor guide
Assessment
This issue has not been assessed yet.