[Discuss] Metrics Plugin for Cross Cluster Search

Is your feature request related to a problem? Please describe.
Currently there is no way to track the cross cluster request/connection metrics. The connection from local to remote cluster is tracked in the local cluster using the remote cluster alias provided in the settings. But there are no metrics tracked along with the connection. We would like APIs to obtain the following information:

• Total number of outbound connections from the source cluster
• Total number of inbound connections to the destination cluster
• Total number of outgoing cross cluster search requests from source cluster
• Total number of incoming cross cluster search requests to destination cluster

Describe the solution you'd like
A plugin to provide the above APIs would solve the problem.
Source cluster:
The total connections can be tracked when the cluster settings are added to create a new connection to the remote cluster.
The outgoing requests can be tracked using an interceptor which will aggregate the count using the remote cluster alias as the key. The remote cluster alias will be unique at the local cluster level.
Destination cluster:
Tracking the metrics at the destination cluster is tricky as there is no setting that is updated here. So it is difficult to track the total number of connections. Also, even if we set the connection alias at the destination cluster, it may conflict with other aliases. For example, cluster A can create remote1 alias for connection cluster B. But cluster A cannot use remote1 again for connection to cluster C. Although cluster C can create a connection remote1 to cluster B. So tracking the number of requests at the destination cluster needs some unique identifier.

Describe alternatives you've considered
One way to track metrics at the destination cluster is to use unique connection id for each combination of clusters and add this setting at both the clusters. This would require additional changes in the core cross cluster search feature to support connection ids.
Looking for more ideas to solve this problem.

Additional context
Add any other context or screenshots about the feature request here.

No files or tests are named. Start by reviewing the core cross cluster search connection settings and the request interceptor described in the issue, then compare source- and destination-cluster metric requirements. Done requires an agreed design for the plugin APIs and a unique way to identify destination-side connections and requests.