pingcap / pingcap/tidb

br: reduce memory footprint and HWM (high water mark)

Open
#51,573 1 comment 0 reactions 0 assignees View on GitHub
component/br type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
BR can consume large size of volume especially at SAAS scenarios due to huge number of db/tables, temporary cache etc.
In fact, most of those memory is one time usage and can be allocated in smaller batches and also be released sooner.
In this issue, we track improvement on this area.

## Common
* https://github.com/pingcap/tidb/pull/51082

The GC memory limit tuner would adjust the golang GC memory limit to a value close to TiDB server environment memory instead of BR's. Besides, backup/restore is a task with a lot of temporary memory, which requires to trigger GC frequently. Therefore, PR#51082 forbidden the GC memory limit tuner in BR binary.

* https://github.com/pingcap/tidb/issues/52781

Make stats export/import under DXF.

* https://github.com/pingcap/tidb/pull/52741

Catch possible goroutine leak

* https://github.com/pingcap/tidb/pull/52741

Automatically adjust `GOMEMLIMIT` for br clp

* https://github.com/pingcap/tidb/pull/53793

## Backup
* https://github.com/pingcap/tidb/pull/43003
* https://github.com/pingcap/tidb/pull/47114
* https://github.com/pingcap/tidb/pull/58591
* https://github.com/pingcap/tidb/pull/60174

Before v7.1.0, when the upstream cluster had a large number of wide tables, it was possible for BR to consume a lot of memory during the backup process. During a backup process, BR would keep three copies of the table information in memory:
1. The InfoSchema maintained by the background domain.
2. The information of the databases and tables being prepared for backup.
3. The serialized schema information before uploading them to external storage.

PR#43003 removes the aforementioned second point of table/databse information. Instead, it adopts a traversal execution approach to promptly release the memory of information of backed up tables.

PR#47114 removes the aforementioned third point of table/databse information. It saves the schema information into some files, and the size of each file is at most 128 MB.

PR#58591 optimizes the memory usage of auxiliary structure of incomplete regions.

PR#60174
- release ranges in time.
- flush the backup response to reduce the response memory consumption.
- use the shared btree.FreeListG for range trees.
- maintain the checksum during backup ranges instead of loading metafiles from external storage.

For the aforementioned first point, we will use BRIE via SQL on TiDB in future, and the TiDB shares the domain with BR task.

## Restore
* https://github.com/pingcap/tidb/pull/49628
* https://github.com/pingcap/tidb/pull/49973
* https://github.com/pingcap/tidb/pull/57192

There might be a table with a large size of statistics (sometimes the table has many partitions). BR uses a lot of memory when backup/restore the table.

PR#49973 supports to dump/load statistics in partition dimension.
PR#49628 supports for BR to persist/restore the statistics data in partition dimension.
PR#57192 prevent preallocating too much items and uses too much memory.

## Log Task
* https://github.com/pingcap/tidb/pull/52127

There is no need to start domain for br log operation except log restore. PR#52127 stops to start domain and creates etcd client by br itself.

* https://github.com/pingcap/tidb/pull/52594
Make sure connection to TiKV stores all closed finally

## BR in SQL
* https://github.com/pingcap/tidb/issues/48046

Put BR in SQL under the memory quota control framework

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.