apple / apple/foundationdb

Cluster replica configuration issue causing cluster unavailability

Open
#10,890 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Hi guys, I encountered an issue where I understand that the configuration is incorrect, but it shouldn't cause the FoundationDB cluster to become abnormal.

FoundationDB Version: 7.1.23

Issue description:

1. Set up a FoundationDB cluster with two nodes. Data can be read and written at this point; for example, set k v.
2. Modify the replicas using the command "configure double." This command times out and fails.
3. Try to get k. Currently, k cannot be read(), but the cluster status is ok.

```shell
fdb> writemode on
fdb> set k v
Committed (186295280)
fdb> get k
`k' is `v'
fdb> configure double
WARNING: Long delay (Ctrl-C to interrupt)
The database is available.
ERROR: The database is unavailable
Type `configure FORCE ' to configure without this check
fdb> status

Using cluster file `/var/fdb/fdb.cluster'.

Configuration:
Redundancy mode - double
Storage engine - memory-2
Coordinators - 1
Usable Regions - 1

Cluster:
FoundationDB processes - 2
Zones - 2
Machines - 2
Memory availability - 8.0 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 09/12/23 10:01:19

Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 0 MB
Disk space used - 210 MB

Operating space:
Storage server - 1.0 GB free on most full server
Log server - 493.7 GB free on most full server

Workload:
Read rate - 15 Hz
Write rate - 0 Hz
Transactions started - 6 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz

Backup and DR:
Running backups - 0
Running DRs - 0

Client time: 09/12/23 10:01:19
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two-node setup through the fdb shell, then run `configure double`, `get k`, and `status` using the reported cluster file `/var/fdb/fdb.cluster`. Done means determining why the failed replica configuration leaves reads unavailable despite the reported status, and verifying the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.