pingcap / pingcap/tidb

Maximum number of range partition tables tested

Open
#34,011 4 comments 0 reactions 1 assignee Claimed by @bb7133 View on GitHub
component/tablepartition component/tiflash severity/minor sig/execution type/bug type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
Reproduce by running this tool with the following command:
[CMD] go-ycsb load mysql -P workloads/oncall2929 -p mysql.host=* -p mysql.port=* -p threadcount=10 -p operationcount=2000 -p recordcount=0 -p initialpartitioncount=10000 -p autommit=0 -p databasename=partition360 -p tablecount=1 -p tablenameprefix=partition361_ -p droppartitioninterval=1000
- param: initialpartitioncount -> create 10000 partition
tablecount -> create # tables in db
other parameters are not important for this issue.
[tool] the tool is customized by this repo: https://github.com/yongpan0709/workloads, but the name 'go-ycsb' is reserved.

The cluster size:
TiDB * 2
TiKV * 4

image

```
CREATE TABLE `partition361_0` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',
`m_req_all` bigint(20) unsigned NOT NULL DEFAULT '0',
`text` varchar(2048) NOT NULL DEFAULT '',
`create_date` date NOT NULL DEFAULT '2000-01-01',
`aid` int(11) unsigned NOT NULL DEFAULT '0',
`atime` time NOT NULL DEFAULT '-01:00:00',
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`,`create_date`) /*T![clustered_index] NONCLUSTERED */,
UNIQUE KEY `unq_p` (`aid`,`atime`,`create_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
PARTITION BY RANGE COLUMNS(`create_date`)
```

Add 8192 partition tables to this partition361_0 【Maximum number of partitions. The maximum possible number of partitions for a given table not using the [NDB](https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster.html) storage engine is 8192. This number includes subpartitions. // https://github.com/pingcap/tidb/pull/20089】

`ALTER TABLE partition360.partition361_0 ADD PARTITION ( PARTITION p20350127 VALUES LESS THAN ('20350128') );
`

The only DDL operations for this cluster load are adding partitioned tables

image

### 2. What did you expect to see? (Required)
the cost of creating one partition < 1s under a strict limit on maximum number of partitions per table.

### 3. What did you see instead (Required)
Before hitting the limit, the cost is too large:
After 20 minutes, the time to execute a single DDL operation rose to 5 seconds

image

After a few hours, the single DDL execution time went up to 20 seconds, and after 7,000 executions it went up to 30 seconds

image

image

Flame map captured during the execution

image
image

### 4. What is your TiDB version? (Required)

```
v6.1.0-nightly
Edition: Community
Git Commit Hash: d3e076c2424c16329786a5433dfe1a27985eb113
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-04-13 23:03:34
GoVersion: go1.18
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.