apple / apple/foundationdb

Better explanation of recovery stall due to policy

Open
#3,876 0 comments 0 reactions 0 assignees View on GitHub
diagnostics
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We had a report that someone configure a multi-region cluster with satellites and had the cluster fail to recover after excluding the satellite processes. This is the status reported by `fdbcli`:

```
db> status details
Using cluster file `fdb-primary.cluster’.
Recruiting new transaction servers.
Need at least 2 log servers across unique zones, 1 commit proxies, 1 GRV
proxies and 1 resolvers.

Have 8 non-excluded processes on 8 machines across 8 zones.
```

This error message is implying that we don't have the processes required to meet our policy (which is correct), but from the data provided it would appear that things should work. This error message should be made more clear to explain what is missing.

See https://forums.foundationdb.org/t/how-to-recover-fdb-database-from-attempt-of-excluding-the-single-sattellite-node/2381.

Contributor guide

Open the contributing guide

Research direction

Start with fdbcli and the `status details` output shown in the issue, then compare the reported non-excluded processes with the policy requirements and the linked forum report. Done means the status explains which policy requirement remains unsatisfied in this multi-region satellite-exclusion scenario, with coverage for the revised wording.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.