Reduce compilation overhead which is introduced by fine grained stream shuffle optimization
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
For MPP queries with agg, join and window functions, we can apply FineGrained stream optimization to improve performance. However, it needs all tiflash nodes' cpu info to determine a proper stream count.
In https://github.com/pingcap/tidb/pull/40121, we query the store server info during compilation, and automatically determine the stream count. But it also introduces compilation latency, in milliseconds, some cases will be 20 milliseconds. When users run low-cardinary agg or small data sets, the total execution time will be dozens of milliseconds, thus the compilation overhead will be significant.
Currently, we can set tiflash_fine_grained_shuffle_stream_count=-1 to close the optimization.
Contributor guide
Assessment
This issue has not been assessed yet.