Cluster replica configuration issue causing cluster unavailability
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Hi guys, I encountered an issue where I understand that the configuration is incorrect, but it shouldn't cause the FoundationDB cluster to become abnormal.
FoundationDB Version: 7.1.23
Issue description:
1. Set up a FoundationDB cluster with two nodes. Data can be read and written at this point; for example, set k v.
2. Modify the replicas using the command "configure double." This command times out and fails.
3. Try to get k. Currently, k cannot be read(), but the cluster status is ok.
```shell
fdb> writemode on
fdb> set k v
Committed (186295280)
fdb> get k
`k' is `v'
fdb> configure double
WARNING: Long delay (Ctrl-C to interrupt)
The database is available.
ERROR: The database is unavailable
Type `configure FORCE ' to configure without this check
fdb> status
Using cluster file `/var/fdb/fdb.cluster'.
Configuration:
Redundancy mode - double
Storage engine - memory-2
Coordinators - 1
Usable Regions - 1
Cluster:
FoundationDB processes - 2
Zones - 2
Machines - 2
Memory availability - 8.0 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 09/12/23 10:01:19
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 210 MB
Operating space:
Storage server - 1.0 GB free on most full server
Log server - 493.7 GB free on most full server
Workload:
Read rate - 15 Hz
Write rate - 0 Hz
Transactions started - 6 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 09/12/23 10:01:19
```
Contributor guide
Research direction
Start by reproducing the two-node setup through the fdb shell, then run `configure double`, `get k`, and `status` using the reported cluster file `/var/fdb/fdb.cluster`. Done means determining why the failed replica configuration leaves reads unavailable despite the reported status, and verifying the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100