pingcap / pingcap/tiflash

Memory usage raise about 4 GB when there are stale snapshot

Open
#5,084 8 comments 0 reactions 1 assignee View on GitHub

@hehechen is already working on this.

Since Jul 18, 2022.

component/storage severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
# build and patch tiflash with failpoint enabled release version
> cmake -DCMAKE_BUILD_TYPE=RELWITHDBGINFO -DENABLE_FAILPOINTS=ON -GNinja ..
# use br to load chbenchmark 1500 from internal minio service

# enable failpoint for holding the snapshot for a while
> LD_LIBRARY_PATH=. ./tiflash client --host ${tiflash_ip} --port ${tiflash_tcp_port} --query 'DBGInvoke enable_fail_point(force_slow_page_storage_snapshot_release)'
> LD_LIBRARY_PATH=. ./tiflash client --host ${tiflash_ip} --port ${tiflash_tcp_port} --query 'selraw count(*) from db_69.t_79'
471415523

https://github.com/pingcap/tiflash/blob/e3a44124f17482bab9039f1b60fc275fe5acfbf2/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp#L1139-L1145

2. What did you expect to see? (Required)

After the snapshot created, the memory usage do no change significantly

3. What did you see instead (Required)

After the snapshot created, the memory usage raise about 4 GB, without any write operations.
After the snapshot released, the memory usage will drop to normal level.

http://172.16.5.81:21510/d/SVbh2xUWk/jayson-tiflash-summary?orgId=1&from=1654597201960&to=1654599606104
image

4. What is your TiFlash version? (Required)

https://github.com/JaySon-Huang/tiflash/commits/debug_ps base on e3a44124f17482bab9039f1b60fc275fe5acfbf2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.