pingcap / pingcap/tidb

statistics: add concurrency for async merge global stats to improve performance

Open
#59,656 1 comment 0 reactions 1 assignee Claimed by @L-maple View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
### Background
In the new TiDB version, `async merge global stats` feature is supported to reduce memory usage and merge stats time.

Async merge global stats is really important for our business clusters, because more than 100+ partitioned tables exist and each partition is partitioned by day range. The largest number of partition now is about 2200+.

But in my sysbench test, merging 4096 partition stats takes about **8 mins** and it brings some risks for online.

### Optimization
parallel merge fmsketch/cmsketch/histogramAndTopN, so that the performance of merging 4096 partition stats improves about 30%.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.