Try optimize the read performance when all packs are filtered
Open
Nobody has claimed this yet.
type/enhancement
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
In a testing env, we found that even if all packs can be filtered by rough set index, the query still takes about 40~50 ms. It mainly comes from building the read buffer when constructing the inputstream of Segments.
It is OK for most queries that will actually read data. However, there is room for optimization for the short queries that can almost filter all data.
-- optimized by generated store column in TiFlash, and use info logging level,
-- takes about 40~50ms
mysql> explain analyze SELECT
-> count(*)
-> FROM
-> token_transfers_new
-> WHERE
-> `diff_bn` <= 0
-> GROUP BY
-> next_block_number;
+------------------------------+---------+---------+--------------+---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+------------------------------+---------+---------+--------------+---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| TableReader_44 | 1.00 | 0 | root | | time:52.4ms, loops:1, RU:0.000000, cop_task: {num: 1, max: 0s, proc_keys: 0, copr_cache_hit_ratio: 0.00} | MppVersion: 2, data:ExchangeSender_43 | 486 Bytes | N/A |
| └─ExchangeSender_43 | 1.00 | 0 | mpp[tiflash] | | tiflash_task:{time:50.9ms, loops:0, threads:36} | ExchangeType: PassThrough | N/A | N/A |
| └─Projection_38 | 1.00 | 0 | mpp[tiflash] | | tiflash_task:{time:50.9ms, loops:0, threads:36} | Column#11 | N/A | N/A |
| └─HashAgg_36 | 1.00 | 0 | mpp[tiflash] | | tiflash_task:{time:50.9ms, loops:0, threads:36} | group by:web3bench.token_transfers_new.next_block_number, funcs:count(1)->Column#11, stream_count: 36 | N/A | N/A |
| └─ExchangeReceiver_21 | 0.00 | 0 | mpp[tiflash] | | tiflash_task:{time:50.9ms, loops:0, threads:36} | stream_count: 36 | N/A | N/A |
| └─ExchangeSender_20 | 0.00 | 0 | mpp[tiflash] | | tiflash_task:{time:59.3ms, loops:0, threads:72} | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: web3bench.token_transfers_new.next_block_number, collate: N/A], stream_count: 36 | N/A | N/A |
| └─TableFullScan_18 | 0.00 | 0 | mpp[tiflash] | table:token_transfers_new | tiflash_task:{time:58.3ms, loops:0, threads:72}, tiflash_scan:{dtfile:{total_scanned_packs:0, total_skipped_packs:13427, total_scanned_rows:0, total_skipped_rows:108000000, total_rs_index_check_time: 5ms, total_read_time: 0ms, total_disagg_read_cache_hit_size: 0, total_disagg_read_cache_miss_size: 0}, total_create_snapshot_time: 0ms, total_local_region_num: 320, total_remote_region_num: 0, total_learner_read_time: 5ms} | pushed down filter:le(web3bench.token_transfers_new.diff_bn, 0), keep order:false | N/A | N/A |
+------------------------------+---------+---------+--------------+---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
7 rows in set
Time: 0.081s
Related codes
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the read-buffer construction for Segment input streams and the related code and discussion in PR #8757. Reproduce the provided TiFlash query in a testing environment, then measure the all-packs-filtered path; done means reducing its unnecessary read-buffer overhead while preserving query behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100