apple / apple/foundationdb

Rethink worker's behavior at io_error

Open
#3,269 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

PR https://github.com/apple/foundationdb/pull/3268 fixes a problem that io_error on shared_tlogs causes repeatable master recovery.

Should we consider killing SS workers on io_error as well? or maybe
we can degrade a stateful process to a stateless process?

A.J.'s comment on why the existing behavior was chosen:

>There is some thought that the reason the existing behavior was chosen in the first place was to prevent us from immediately trying to reuse a storage disk after it gets an IO error. The current code doesn't completely avoid that possibility, since the process could restart for a variety of reasons, but in many cases an IO error will leave the process running but not operating as a storage server. That generally will give you some time to do replace the machine or take some other action in response to a disk failure.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR 3268 and the issue's explanation of current io_error behavior for shared_tlogs and SS workers. The scope is not yet settled: completion requires an agreed decision between killing SS workers, degrading them to stateless processes, or retaining the current behavior, followed by an implementation and validation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.