pingcap / pingcap/tidb

Reduce compilation overhead which is introduced by fine grained stream shuffle optimization

Open
#45,053 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
For MPP queries with agg, join and window functions, we can apply FineGrained stream optimization to improve performance. However, it needs all tiflash nodes' cpu info to determine a proper stream count.
In https://github.com/pingcap/tidb/pull/40121, we query the store server info during compilation, and automatically determine the stream count. But it also introduces compilation latency, in milliseconds, some cases will be 20 milliseconds. When users run low-cardinary agg or small data sets, the total execution time will be dozens of milliseconds, thus the compilation overhead will be significant.
Currently, we can set tiflash_fine_grained_shuffle_stream_count=-1 to close the optimization.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.