pingcap / pingcap/tiflash

tiflash-server failed to start when tiflash-server is crash

Open
#4,727 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/unknown severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

step one:tiflash store size 1TB
step two:tiflash memory size 32GB
step three:restart tiflash-server

2. What did you expect to see? (Required)

tiflash-server always starts normally, when its status is down

3. What did you see instead (Required)

Since the value of alwaysRestart is true, tiflash-server starts automatically every time it is oom-killer by OS, which causes the server memory to be full and the login account is slow to execute commands and cannot do other debugging.

After I increase the memory to make tiflash-server start, when no command is run to tiflash, it only uses 5GB of memory, which is a waste of resources

In the initialization node of tiflash-server, the throughput of the disk is 350MiB. I am the disk of io1. This needs to be paid attention to. It really takes up a lot of disk traffic.

He may not save our cost in the tidb-cloud environment

4. What is your TiFlash version? (Required)

v5.4.0

5. some feature
  1. Reduced memory required for tiflash-server startup
  2. Possibly a resource control for the initialization of the tiflash-server at startup, adding a tunable concurrency parameter (pool-size)? Or a flow-control parameter?
  3. During troubleshooting, we can only diagnose my problem through the memory monitoring in the tiflash-summary monitoring. It may be necessary to add a load phase monitoring to help us know what is wrong with our tiflash-server. (Not to mention tiflash.log, the debug-level parameters are still not helpful for the current investigation, and they take up a lot of disk io.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no files or tests; start by reproducing the startup failure on TiFlash v5.4.0 with the stated 1 TB store and 32 GB memory, then inspect the initialization path and the memory, disk-throughput, and load monitoring discussed in the report. Done should be a scoped change with reproducible validation and clear acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.