Microcluster itself can be out of sync with the underlying Dqlite’s membership configuration.

I tested clustering with Canonical k8s and embedded etcd.

Hereby, I noticed that Microcluster itself can be out of sync with the underlying Dqlite’s membership configuration. In this scenario we have a three node cluster where we take down one node by killing its VM process. Then we join a node and try to remove the previously killed node. We observe diverging cluster state: Microcluster table shows 3 nodes as voters, Microcluster’s Dqlite has 4 nodes with the killed one marked as a spare and etcd shows 2 nodes.

See also: https://github.com/canonical/go-dqlite/issues/388, https://github.com/canonical/dqlite/issues/799