Rethink worker's behavior at io_error
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
PR https://github.com/apple/foundationdb/pull/3268 fixes a problem that io_error on shared_tlogs causes repeatable master recovery.
Should we consider killing SS workers on io_error as well? or maybe
we can degrade a stateful process to a stateless process?
A.J.'s comment on why the existing behavior was chosen:
>There is some thought that the reason the existing behavior was chosen in the first place was to prevent us from immediately trying to reuse a storage disk after it gets an IO error. The current code doesn't completely avoid that possibility, since the process could restart for a variety of reasons, but in many cases an IO error will leave the process running but not operating as a storage server. That generally will give you some time to do replace the machine or take some other action in response to a disk failure.
Contributor guide
Research direction
Start by reviewing PR 3268 and the issue's explanation of current io_error behavior for shared_tlogs and SS workers. The scope is not yet settled: completion requires an agreed decision between killing SS workers, degrading them to stateless processes, or retaining the current behavior, followed by an implementation and validation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100