Disconnecting all network modules because of one bad subnet is disrupting all communication

This issue is encountered when running a comdb2 cluster with two or more physical networks between nodes (a.k.a subnets). The use of multiple subnets tries to make the cluster segmentation a less probably event, so we can forget about CAP for a minute.

Problem: When any of the subnets has a problem, the server disconnects the other modules, and this gets replicated to the other nodes; in effect, all sockets to all nodes are dropped, disrupting the cluster.

Details:
The comdb2 server has 3 peer-2-peer networks in the cluster: replication, offloadsql, and signal (the last should go soon). Lets call them R, O, S for the sake of example. Lets say nodes are connecting using N1 and N2 physical networks, i.e. the subnets. Lets say the cluster has 3 nodes A, B, C. Basically, node A uses 3 sockets to connect to B, and 3 more to connect to C, and each socket can be on either N1 or N2, randomly (same for B and C).
Example:
Lets assume A has R(N1, B) (R uses N1 to connect to B), O(N2, B), S(N2, B), R(N2,C), O(N1, C), S(N1, C);
B has R (N2,C), O(N1,C), S(N2,C); of course the reverse is true, sockets are full-duplex.

Lets say that N1 has a problem, which kills all the connections, or worse some sockets that use N1. The socket connections that fail on N1 should be reopened on N2 in a proper system.

THis is what happens initially:
(1)A swings R(N1, B) to R(N2, B).
(2) A swings O(N1, C) to O(N2, C)
(3) A swings S(N1, C) t0 S(N2, C)
(4) B also swings O(N1,C) to O(N2,C)

Now, this should be great, except that server is disconnecting for each event the rest of the R, O or S modules, too. If either R, S or O is already swinging, the additional swing is skipped, otherwise the socket is shutdown again and reopened the old subnet.
So:
(5) swing (1) involving R, will disconnect and swing O and S from N2 to N1 (back to bad)!
(6) swing (2) also swings again R and S; if (5) is in progress, this is nop, otherwise R is moved again from N2 to N1 (back to bad).
And so on. If you tried to follow so far, kudos to you; a disconnect on one side gets replicated in disconnects everywhere else, and the cluster is for awhile just a set of disconnected nodes.

Initial Proposal: kill the disconnecting of additional networks logic; instead of disconnect; we should communicate to higher levels single subnet failures so that any recovery is performed (for example; dropping offloadsql network would leave replication alone, but would tell sqlite engine to check the session and eventually retransmit the bplog to the master).

Example from testing R7 on our 2 subnet cluster of 3 nodes: shutting down sockets on n4 once, this will kill sockets on n3 too, multiple times:
0xedcad700 [offloadsql pnj-comdb2alpo2-n3 fd 14] shutting down fd 14
0xedcad700 [replication pnj-comdb2alpo2-n3 fd 270] shutting down fd 270
0xefd3b700 [offloadsql pnj-comdb2alpo1-n4 fd 12] shutting down fd 12
0xedcad700 [signal pnj-comdb2alpo2-n4 fd 269] shutting down fd 269
0xefd3b700 [replication pnj-comdb2alpo1-n3 fd 19] shutting down fd 19
0xefd3b700 [signal pnj-comdb2alpo1-n4 fd 18] shutting down fd 18
0xf0a41700 [replication pnj-comdb2alpo2-n3 fd 270] shutting down fd 270
0xf0a41700 [offloadsql pnj-comdb2alpo2-n3 fd 14] shutting down fd 14
0xedc6c700 [offloadsql pnj-comdb2alpo1-n4 fd 12] shutting down fd 12
0xf0a82700 [offloadsql pnj-comdb2alpo1-n4 fd 12] shutting down fd 12
0xedc6c700 [replication pnj-comdb2alpo1-n3 fd 19] shutting down fd 19