apple / apple/foundationdb

Optimize failure reaction times

Open
#1,039 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The failure monitor waits at least 4 seconds before declaring a process as dead. The reason this delay is set so high is because of the possibility of slow tasks, which could prevent a process from notifying the failure monitor that it is still alive.

After we track down and eliminate long slow tasks we should be able to lower this reaction time.

The slow tasks on the storage server caused by SQLite might not be fixable, in which case we should use role information to selectively increase the reaction time on processes running the SQLite storage engine.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the failure monitor's four-second threshold and the slow-task paths involving the SQLite storage server. Identify which long tasks delay liveness notifications and determine whether they can be eliminated or require role-specific reaction times. Done means the reaction time is safely reduced or selectively increased for SQLite processes, with validation of failure detection behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sqlite
Domain
databases, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.