tikv / tikv/pd

Keyspace related HTTP API may encounter error or performance issue when there are too many keyspaces

Open
#10,736 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

* Create 100K keyspaces by posting to `/pd/api/v2/keyspaces` HTTP API
* Get keyspace meta by getting from `/pd/api/v2/keyspaces` HTTP API
* Iterate all keyspaces in PD's code

### What did you expect to see?

Everything works fine

### What did you see instead?

* Create 100K keyspaces by posting to `/pd/api/v2/keyspaces` HTTP API
* With 10 concurrency in the client, the performace slows down as the progress proceeds, and it takes about 7-8 hours to finish creating all the keyspaces.
* The reason might be the `UpdateKeyspaceForGroup` step.
* Get keyspace meta by getting from `/pd/api/v2/keyspaces` HTTP API
* ```
$ curl 192.168.178.11:3250/pd/api/v2/keyspaces
"rpc error: code = ResourceExhausted desc = trying to send message larger than max (11983608 vs. 2097152)"
```
* Iterate all keyspaces in PD's code
* It takes about 90s to finish iterating 100K keyspaces.

### What version of PD are you using (`pd-server -V`)?

master, with unrelated modifications in `GCStateManager` (see: https://github.com/tikv/pd/pull/10498)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.