tiflash shows huger total_scanned_rows when query to new cluster just created with only inserted data
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
create table log_link_visit_action(idvisit varchar(100), idvisitor varchar(100), time_dom_completion int, time_on_load int);
alter table log_link_visit_action set tiflash replica 1;
insert into log_link_visit_action values('1', '2', 1,1);
insert into log_link_visit_action values('1', '3', 1,1);
insert into log_link_visit_action values('1', '4', 1,1);
insert into log_link_visit_action select * from log_link_visit_action;
=> Util we get almost 1258,2912 rows
2. What did you expect to see? (Required)
explain analyze select count(distinct idvisit), count(distinct idvisitor), count(time_dom_completion), sum(time_on_load) from log_link_visit_action;
total_scanned_rows:15616550 should be exactly equal to 1258,2912 rows
Even with another insert.script to import the same 1258,2912 rows, the total_scanned_rows shows even more about it
3. What did you see instead (Required)
MySQL [test]> explain analyze select count(distinct idvisit), count(distinct idvisitor), count(time_dom_completion), sum(time_on_load) from log_link_visit_action;
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| id | estRows | actRows | task | access object | execution info | operator info | memory | disk |
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| TableReader_30 | 1.00 | 1 | root | | time:277.1ms, loops:2, cop_task: {num: 2, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00} | MppVersion: 1, data:ExchangeSender_29 | 537 Bytes | N/A |
| └─ExchangeSender_29 | 1.00 | 1 | mpp[tiflash] | | tiflash_task:{time:275.3ms, loops:1, threads:20} | ExchangeType: PassThrough | N/A | N/A |
| └─Projection_25 | 1.00 | 1 | mpp[tiflash] | | tiflash_task:{time:275.3ms, loops:1, threads:20} | Column#6, Column#7, Column#8, Column#9 | N/A | N/A |
| └─HashAgg_26 | 1.00 | 1 | mpp[tiflash] | | tiflash_task:{time:275.3ms, loops:1, threads:20} | funcs:count(distinct test.log_link_visit_action.idvisit)->Column#6, funcs:count(distinct test.log_link_visit_action.idvisitor)->Column#7, funcs:sum(Column#12)->Column#8, funcs:sum(Column#13)->Column#9 | N/A | N/A |
| └─ExchangeReceiver_28 | 1.00 | 3 | mpp[tiflash] | | tiflash_task:{time:274.3ms, loops:1, threads:20} | | N/A | N/A |
| └─ExchangeSender_27 | 1.00 | 3 | mpp[tiflash] | | tiflash_task:{time:273ms, loops:1, threads:12} | ExchangeType: PassThrough | N/A | N/A |
| └─HashAgg_24 | 1.00 | 3 | mpp[tiflash] | | tiflash_task:{time:273ms, loops:1, threads:12} | group by:Column#16, Column#17, funcs:count(Column#14)->Column#12, funcs:sum(Column#15)->Column#13 | N/A | N/A |
| └─Projection_31 | 12582912.00 | 12582912 | mpp[tiflash] | | tiflash_task:{time:23ms, loops:215, threads:12} | test.log_link_visit_action.time_dom_completion, cast(test.log_link_visit_action.time_on_load, decimal(10,0) BINARY)->Column#15, test.log_link_visit_action.idvisit, test.log_link_visit_action.idvisitor | N/A | N/A |
| └─TableFullScan_13 | 12582912.00 | 12582912 | mpp[tiflash] | table:log_link_visit_action | tiflash_task:{time:16ms, loops:215, threads:12}, tiflash_scan:{dtfile:{total_scanned_packs:1917, total_skipped_packs:0, total_scanned_rows:15616550, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 637ms}, total_create_snapshot_time: 0ms, total_local_region_num: 15, total_remote_region_num: 0} | keep order:false | N/A | N/A |
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
9 rows in set (0.28 sec)
4. What is your TiFlash version? (Required)
master
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the EXPLAIN ANALYZE query and compare the actual row count with the TiFlash TableFullScan total_scanned_rows metric. Review the reported tiflash_scan details and the linked insert.go script to isolate whether newly inserted data is being scanned more than once. Done means the scan metric matches the rows present in the table for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- analytics, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100