Fix NiFi 2 clustering

Description

Currently, if you create a JWT token for a nifi node, you can only log into this specific node. This is apparently by design as explained in https://issues.apache.org/jira/browse/NIFI-7246.

The problem with this is that it forces sticky sessions (when using a load balancer) which defeats the purpose of using a stateless token.

This appears to exist because some NiFi state is not shares, so for some API calls you currently must connect to the applicable node.

Slightly relevant is that there are SNI checks (so that the server can verify that the client is connecting via the expected hostname). Presumably this was implemented due to the previously mentioned requirement for connecting to the correct node.

Proposed Solution

[!TIP]
As this is a specific problem of the product NiFi we think it might be useful to fix it upstream.

There are multiple parts to the overall solution:

1. Fix NiFi 2's state sharing so that there is no longer a need to send requests to a specific node.
2. Allow JWTs to be issued by a cluster wide issuer for a single audience so that the same token can be used on any node (thereby removing the need for sticky sessions).
3. Additional nice-to-have's:
   • Remove the SNI hostname check (assuming it was only there as a protection mechanism for what was mentioned in the Description section).
   • Possibly remove authentication from metrics endpoints (and ideally run metrics on a different http server).

Unknowns

• How is the JWT signing key currently generated?
• Can the SNI check be removed once it doesn't matter which cluster node is connected to?

Start by investigating NiFi 2's state sharing and cluster-wide JWT behavior, including how the signing key is generated and how SNI hostname checks are used. Also review the metrics authentication path. Done means cluster-issued tokens work on any node without sticky sessions, with the proposed SNI and metrics changes evaluated.