apache / apache/pekko

Clustering issues leading to all nodes being downed

Open
#578 25 comments 1 reaction 0 assignees View on GitHub
bug help wanted needs-reproducible-test
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

I'm reopening here an [issue that I reported at the time under the akka repo](https://github.com/akka/akka/issues/31095).

We had a case where an issue on a single node lead to the whole akka-cluster being taken down.

### Here's a summary of what happened:
1. Healthy cluster made of 20ish nodes, running on k8s
2. Node A: encounters issues, triggers CoordinatedShutdown
3. Node A: experiences high CPU usage, maybe GC pause
4. Node A: sees B as unreachable, broadcasts it (B is certainly reachable, but detected as such because of high CPU usage, GC pause, or similar issues)
5. Cluster state: A Leaving, B seen unreachable by A, all the other nodes are Up
6. Leader can currently not perform its duties (remove A), reachability status (B seen unreachable by A)
7. Node A: times out some coordinated shutdown phases. Hypothesis: timed out because leader could not remove A.
8. Node A: finishes coordinated shutdown nonetheless.
9. hypothesis - Node A: quarantined associations to other cluster nodes
10. Nodes B, C, D, E: SBR took decision DownSelfQuarantinedByRemote and is downing [...] including myself
11. hypothesis - Node B, C, D, E: quarantined associations to other cluster nodes
12. in a few steps, all remaining cluster nodes down themselves: SBR took decision DownSelfQuarantinedByRemote
13. the whole cluster is down

### Discussions, potential issues:

Considering the behaviour of CoordinatedShutdown (phases can time out and shutdown continues), shouldn't the leader ignore unreachabilities added by a Leaving node and be allowed to perform its duties?
At step 6 above, the Leader was blocked from removing A, but A still continued its shutdown process. The catastrophic ending could have been stopped here.

DownSelfQuarantinedByRemote: @patriknw 's [comment](https://github.com/akka/akka/pull/29737#discussion_r515906571) seems spot on.
At step 9, nodes B, C, D, E should probably not take into account the `Quarantined` from a node that is Leaving.

DownSelfQuarantinedByRemote: another case where Patrik's [comment](https://github.com/akka/akka/pull/29737#discussion_r515906571) also seems to apply, `Quarantined` from nodes downing themselves because of DownSelfQuarantinedByRemote should probably not be taken into account.

At steps 10 and 12. Any cluster singletons running on affected nodes wouldn't be gracefully shutdown using the configured termination message. This is probably the right thing to do but I'm adding this note here nonetheless.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Akka issue and the referenced PR #29737 discussion, then trace the CoordinatedShutdown, leader reachability, and DownSelfQuarantinedByRemote sequence described here. The issue needs an agreed reproduction and design before implementation; done should include preventing a single Leaving or self-downing node from cascading the cluster shutdown, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.