Optimize failure reaction times
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The failure monitor waits at least 4 seconds before declaring a process as dead. The reason this delay is set so high is because of the possibility of slow tasks, which could prevent a process from notifying the failure monitor that it is still alive.
After we track down and eliminate long slow tasks we should be able to lower this reaction time.
The slow tasks on the storage server caused by SQLite might not be fixable, in which case we should use role information to selectively increase the reaction time on processes running the SQLite storage engine.
Contributor guide
Research direction
Start by tracing the failure monitor's four-second threshold and the slow-task paths involving the SQLite storage server. Identify which long tasks delay liveness notifications and determine whether they can be eliminated or require role-specific reaction times. Done means the reaction time is safely reduced or selectively increased for SQLite processes, with validation of failure detection behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sqlite
- Domain
- databases, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100