tiflash-server failed to start when tiflash-server is crash
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
step one:tiflash store size 1TB
step two:tiflash memory size 32GB
step three:restart tiflash-server
2. What did you expect to see? (Required)
tiflash-server always starts normally, when its status is down
3. What did you see instead (Required)
Since the value of alwaysRestart is true, tiflash-server starts automatically every time it is oom-killer by OS, which causes the server memory to be full and the login account is slow to execute commands and cannot do other debugging.
After I increase the memory to make tiflash-server start, when no command is run to tiflash, it only uses 5GB of memory, which is a waste of resources
In the initialization node of tiflash-server, the throughput of the disk is 350MiB. I am the disk of io1. This needs to be paid attention to. It really takes up a lot of disk traffic.
He may not save our cost in the tidb-cloud environment
4. What is your TiFlash version? (Required)
v5.4.0
5. some feature
- Reduced memory required for tiflash-server startup
- Possibly a resource control for the initialization of the tiflash-server at startup, adding a tunable concurrency parameter (pool-size)? Or a flow-control parameter?
- During troubleshooting, we can only diagnose my problem through the memory monitoring in the tiflash-summary monitoring. It may be necessary to add a load phase monitoring to help us know what is wrong with our tiflash-server. (Not to mention tiflash.log, the debug-level parameters are still not helpful for the current investigation, and they take up a lot of disk io.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The payload names no files or tests; start by reproducing the startup failure on TiFlash v5.4.0 with the stated 1 TB store and 32 GB memory, then inspect the initialization path and the memory, disk-throughput, and load monitoring discussed in the report. Done should be a scoped change with reproducible validation and clear acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100