br: reduce memory footprint and HWM (high water mark)
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
BR can consume large size of volume especially at SAAS scenarios due to huge number of db/tables, temporary cache etc.
In fact, most of those memory is one time usage and can be allocated in smaller batches and also be released sooner.
In this issue, we track improvement on this area.
## Common
* https://github.com/pingcap/tidb/pull/51082
The GC memory limit tuner would adjust the golang GC memory limit to a value close to TiDB server environment memory instead of BR's. Besides, backup/restore is a task with a lot of temporary memory, which requires to trigger GC frequently. Therefore, PR#51082 forbidden the GC memory limit tuner in BR binary.
* https://github.com/pingcap/tidb/issues/52781
Make stats export/import under DXF.
* https://github.com/pingcap/tidb/pull/52741
Catch possible goroutine leak
* https://github.com/pingcap/tidb/pull/52741
Automatically adjust `GOMEMLIMIT` for br clp
* https://github.com/pingcap/tidb/pull/53793
## Backup
* https://github.com/pingcap/tidb/pull/43003
* https://github.com/pingcap/tidb/pull/47114
* https://github.com/pingcap/tidb/pull/58591
* https://github.com/pingcap/tidb/pull/60174
Before v7.1.0, when the upstream cluster had a large number of wide tables, it was possible for BR to consume a lot of memory during the backup process. During a backup process, BR would keep three copies of the table information in memory:
1. The InfoSchema maintained by the background domain.
2. The information of the databases and tables being prepared for backup.
3. The serialized schema information before uploading them to external storage.
PR#43003 removes the aforementioned second point of table/databse information. Instead, it adopts a traversal execution approach to promptly release the memory of information of backed up tables.
PR#47114 removes the aforementioned third point of table/databse information. It saves the schema information into some files, and the size of each file is at most 128 MB.
PR#58591 optimizes the memory usage of auxiliary structure of incomplete regions.
PR#60174
- release ranges in time.
- flush the backup response to reduce the response memory consumption.
- use the shared btree.FreeListG for range trees.
- maintain the checksum during backup ranges instead of loading metafiles from external storage.
For the aforementioned first point, we will use BRIE via SQL on TiDB in future, and the TiDB shares the domain with BR task.
## Restore
* https://github.com/pingcap/tidb/pull/49628
* https://github.com/pingcap/tidb/pull/49973
* https://github.com/pingcap/tidb/pull/57192
There might be a table with a large size of statistics (sometimes the table has many partitions). BR uses a lot of memory when backup/restore the table.
PR#49973 supports to dump/load statistics in partition dimension.
PR#49628 supports for BR to persist/restore the statistics data in partition dimension.
PR#57192 prevent preallocating too much items and uses too much memory.
## Log Task
* https://github.com/pingcap/tidb/pull/52127
There is no need to start domain for br log operation except log restore. PR#52127 stops to start domain and creates etcd client by br itself.
* https://github.com/pingcap/tidb/pull/52594
Make sure connection to TiKV stores all closed finally
## BR in SQL
* https://github.com/pingcap/tidb/issues/48046
Put BR in SQL under the memory quota control framework
Contributor guide
Assessment
This issue has not been assessed yet.