Support runtime stats for tiflash
Open
@guo-shaoge is already working on this.
Since Feb 24, 2025.
type/enhancement
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
Runtime stats refer to recording the characteristics of a query during execution, and using these statistics to select a more optimal execution strategy the next time the same query runs. Typical runtime stats include:
- Record the average length of the key for HashAgg on key_string. Based on this length, an appropriate HashTable can be chosen. For example, if the average length is greater than 24B, a regular HashTable is used; otherwise, a StringHashTable is selected (by using norma HashTable, we can support prefetch. because StringHashMap doesn't support prefetch) (major, 10%).
- Whether the query last used a two-level HashMap for HashAgg. If so, the two-level HashMap can be used directly in the next execution, saving the time of converting from one-level to two-level. (minor, <=5%)
- Based on the size of the HashTable in HashAgg, an early resize may be performed (though this is harder to implement in TiFlash, as the region selected for a query may change due to load balancing strategies).
- The reserve size for ColumnString. By default, it is assumed that the string length is 64B. If more accurate information becomes available later, it will improve the expression evaluation. (minor, <=5%)
- When the NDV (Number of Distinct Values) in HashAgg is very low, using mini-batch may have side effects. Through runtime stats, the non-mini-batch version can be proactively chosen for the next query execution. (minor, <= 5%)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.