Handle graceful cluster shrinking

Affected version

0.6.0

Current and expected behavior

Currently when a NifiCluster is shrunk in size, the user has to make sure that the node(s) to be removed have been disconnected and drained manually before triggering the shrinking.

Ideally we'd want the operator to use the NiFi Rest api to perform these administrative tasks automatically.

Possible solution

The operator needs to be able to track, whether an action is about to shrink the cluster, determine which pods are going to be removed and trigger the necessary action via the NiFi api for these pods.

In order to do this we'll probably have to track "current pods" in the status of the NifiCluster object, in order for the operator to be able to determine if the new version of the object signifies a shrinking action.
Once this check is triggered, the operator then needs to check if all nodes are drained. If this is the case the reconciliation can proceed normally, if not, drain tasks need to be started for all pods / checked if tasks are already running. Then the operator can requeue, and will simply keep running this check until all tasks are done.

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

No response

Start by reading the NifiCluster reconciliation and status handling to understand how a requested cluster-size change is detected. Trace how pods are selected and how NiFi REST API actions could be awaited across reconciliation; done means removed nodes are disconnected and drained automatically before shrinking proceeds.