apple / apple/foundationdb

FDB cluster with three_datacenter mode becomes unavailable if one of three DCs has network card failure

Open
#8,960 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I have a FDB cluster in three_datacenter mode. Each DC has 6 machines.

I found that the cluster becomes unavailable if I randomly down the machines’ public network interface (ifdown bond1) in the primary DC (fdbcli --exec ‘status json’ | jq .cluster.active_primary_dc).

The problem is hard to recur.

Any advice to address this?

Contributor guide

Open the contributing guide

Research direction

Start by examining the three_datacenter failure scenario described: take down bond1 on machines in the primary DC and query fdbcli with “status json”, checking cluster.active_primary_dc. No source files or tests are identified. Done means producing a reliable reproduction, identifying why the cluster becomes unavailable, and documenting an actionable mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.