pingcap / pingcap/tidb

planner: decrease concurrency / batch_size automatically for streaming plans with limit-clause

Open
#66,658 3 comments 0 reactions 1 assignee Claimed by @windtalker View on GitHub
report/customer sig/execution sig/planner type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
See the example below:

Image

We have a `limit 1000` clause, but TiDB still scanned more than 390000+ rows.
The root cause is that our default concurrency (5, 15) and batch-size (25000) is too large.
We have to scan and process more rows before the plan is stopped by the limit-clause.
This could waste some system resources, like CPU.

After decreasing the concurrency and batch-size via `/*+ set_var(tidb_index_lookup_join_concurrency=2) set_var(tidb_distsql_scan_concurrency=7) set_var(tidb_executor_concurrency=2) set_var(tidb_index_join_batch_size=12000) */`, we can see the CPU peak decreased a lot:

Image

Then for these pure streaming plans that only have streaming (or pipeline) operators like `*Scan`, `IndexJoin`, `StreamAgg`, `MergeJoin` with Limit clause. We can automatically decrease their concurrency and batch-size.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.