pingcap / pingcap/tiflash

tiflash shows huger total_scanned_rows when query to new cluster just created with only inserted data

Open
#6,824 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/storage severity/minor type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
create table log_link_visit_action(idvisit varchar(100), idvisitor varchar(100), time_dom_completion int, time_on_load int);

alter table log_link_visit_action set tiflash replica 1;

insert into log_link_visit_action values('1', '2', 1,1);
insert into log_link_visit_action values('1', '3', 1,1);
insert into log_link_visit_action values('1', '4', 1,1);

insert into log_link_visit_action select * from log_link_visit_action;
=> Util we get almost  1258,2912 rows
2. What did you expect to see? (Required)

explain analyze select count(distinct idvisit), count(distinct idvisitor), count(time_dom_completion), sum(time_on_load) from log_link_visit_action;

total_scanned_rows:15616550 should be exactly equal to 1258,2912 rows

Even with another insert.script to import the same 1258,2912 rows, the total_scanned_rows shows even more about it

3. What did you see instead (Required)
MySQL [test]> explain analyze select count(distinct idvisit), count(distinct idvisitor), count(time_dom_completion), sum(time_on_load) from log_link_visit_action;
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| id                                   | estRows     | actRows  | task         | access object               | execution info                                                                                                                                                                                                                                                                                                              | operator info                                                                                                                                                                                            | memory    | disk |
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
| TableReader_30                       | 1.00        | 1        | root         |                             | time:277.1ms, loops:2, cop_task: {num: 2, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                   | MppVersion: 1, data:ExchangeSender_29                                                                                                                                                                    | 537 Bytes | N/A  |
| └─ExchangeSender_29                  | 1.00        | 1        | mpp[tiflash] |                             | tiflash_task:{time:275.3ms, loops:1, threads:20}                                                                                                                                                                                                                                                                            | ExchangeType: PassThrough                                                                                                                                                                                | N/A       | N/A  |
|   └─Projection_25                    | 1.00        | 1        | mpp[tiflash] |                             | tiflash_task:{time:275.3ms, loops:1, threads:20}                                                                                                                                                                                                                                                                            | Column#6, Column#7, Column#8, Column#9                                                                                                                                                                   | N/A       | N/A  |
|     └─HashAgg_26                     | 1.00        | 1        | mpp[tiflash] |                             | tiflash_task:{time:275.3ms, loops:1, threads:20}                                                                                                                                                                                                                                                                            | funcs:count(distinct test.log_link_visit_action.idvisit)->Column#6, funcs:count(distinct test.log_link_visit_action.idvisitor)->Column#7, funcs:sum(Column#12)->Column#8, funcs:sum(Column#13)->Column#9 | N/A       | N/A  |
|       └─ExchangeReceiver_28          | 1.00        | 3        | mpp[tiflash] |                             | tiflash_task:{time:274.3ms, loops:1, threads:20}                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                          | N/A       | N/A  |
|         └─ExchangeSender_27          | 1.00        | 3        | mpp[tiflash] |                             | tiflash_task:{time:273ms, loops:1, threads:12}                                                                                                                                                                                                                                                                              | ExchangeType: PassThrough                                                                                                                                                                                | N/A       | N/A  |
|           └─HashAgg_24               | 1.00        | 3        | mpp[tiflash] |                             | tiflash_task:{time:273ms, loops:1, threads:12}                                                                                                                                                                                                                                                                              | group by:Column#16, Column#17, funcs:count(Column#14)->Column#12, funcs:sum(Column#15)->Column#13                                                                                                        | N/A       | N/A  |
|             └─Projection_31          | 12582912.00 | 12582912 | mpp[tiflash] |                             | tiflash_task:{time:23ms, loops:215, threads:12}                                                                                                                                                                                                                                                                             | test.log_link_visit_action.time_dom_completion, cast(test.log_link_visit_action.time_on_load, decimal(10,0) BINARY)->Column#15, test.log_link_visit_action.idvisit, test.log_link_visit_action.idvisitor | N/A       | N/A  |
|               └─TableFullScan_13     | 12582912.00 | 12582912 | mpp[tiflash] | table:log_link_visit_action | tiflash_task:{time:16ms, loops:215, threads:12}, tiflash_scan:{dtfile:{total_scanned_packs:1917, total_skipped_packs:0, total_scanned_rows:15616550, total_skipped_rows:0, total_rs_index_load_time: 0ms, total_read_time: 637ms}, total_create_snapshot_time: 0ms, total_local_region_num: 15, total_remote_region_num: 0} | keep order:false                                                                                                                                                                                         | N/A       | N/A  |
+--------------------------------------+-------------+----------+--------------+-----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------+
9 rows in set (0.28 sec)
4. What is your TiFlash version? (Required)

master

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the EXPLAIN ANALYZE query and compare the actual row count with the TiFlash TableFullScan total_scanned_rows metric. Review the reported tiflash_scan details and the linked insert.go script to isolate whether newly inserted data is being scanned more than once. Done means the scan metric matches the rows present in the table for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
analytics, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.