Ensure all state is shared across NiFi 2 cluster members

Part of https://github.com/stackabletech/nifi-operator/issues/809.

Why?

This opens the door for fixing the Oauth2 implementation which solves problems for many (including us).

Current state

Although it is termed a NiFi cluster, cluster members are not exactly replicas, and hold local state.

One example of local state is a Site-to-Site flowfile transfer to another cluster. Currently, to get the status of a transfer, you must connect to the node that you initiated the transfer on.

This might be the reason for the following:

• A JWT issuer per cluster memeber (tokens cannot be used on other cluster members)
• SNI hostname checks

Future state

All cluster state should be shared to cluster members so that cluster members can actually be replicas, and therefore sit behind a load balancer without needing sticky-sessions.

Recommended approach

• Try to find any reason why some state cannot be shared across cluster members.
• Start a conversation with NiFi maintainers to gauge interest in accepting this fix upstream.
• Fork NiFi and try to share the currently unshared state across cluster members.

Unknowns

• Which other state is local only? Perhaps the answer can be found in NIFI-7246

Start by reviewing NIFI-7246 and the linked issues 809 and 811 to identify which NiFi state remains local to cluster members. Then discuss the feasibility with NiFi maintainers and investigate the proposed NiFi fork; done means documenting the blockers or achieving shared state that supports load-balanced members without sticky sessions.