planner: decrease concurrency / batch_size automatically for streaming plans with limit-clause
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
See the example below:
We have a `limit 1000` clause, but TiDB still scanned more than 390000+ rows.
The root cause is that our default concurrency (5, 15) and batch-size (25000) is too large.
We have to scan and process more rows before the plan is stopped by the limit-clause.
This could waste some system resources, like CPU.
After decreasing the concurrency and batch-size via `/*+ set_var(tidb_index_lookup_join_concurrency=2) set_var(tidb_distsql_scan_concurrency=7) set_var(tidb_executor_concurrency=2) set_var(tidb_index_join_batch_size=12000) */`, we can see the CPU peak decreased a lot:
Then for these pure streaming plans that only have streaming (or pipeline) operators like `*Scan`, `IndexJoin`, `StreamAgg`, `MergeJoin` with Limit clause. We can automatically decrease their concurrency and batch-size.
Contributor guide
Assessment
This issue has not been assessed yet.