Tracker becomes non-responsive

The Tracker is getting into a non-responsive state for swarm version 3.0.5 under the following circumstance:

1. The Tracker has messages in the message queue which triggers a broadcast when handled
2. one or more node goes down
3. The handler calls :rpc.sbcast which tries to send a message to all nodes, including the down nodes, and therefore only returns after a timeout to the dead nodes. This continues until the nodedown messages are handled.

Our setup is a kubernetes cluster, where we have observed timeouts of 3-6 seconds before it discovers that a node is down. This makes the Tracker non-responsive until the nodedown message is handled, which potentially takes a lot of time.

A hotfix for this, until it is resolved, could be to call :rpc.abcast instead, since the info about the bad nodes are never really used anyway. Are there any issues with this approach? I can't see that it makes any difference other than missing warnings about the bad nodes.

A fix for this could be to look for nodedown messages in the message queue, when bad nodes are discovered, and then handle nodedown messages accordingly. I'm not sure this is way to do it, just a thought.

Start with the Tracker message handler that calls :rpc.sbcast when a queued message triggers a broadcast, and compare it with :rpc.abcast. Reproduce the issue on Swarm 3.0.5 with a node going down in a Kubernetes cluster, then inspect how nodedown messages are handled. Done means the Tracker remains responsive during node failures without losing required broadcast behavior.

Written by the indexing model from the issue text.