[FEATURE] Index pattern resolution improvements

Overview

Goal of this issue is to implement broad improvements regarding the index pattern resolution in the security plugin.

Index pattern resolution is a vital component of the security plugin; it extracts from action requests metadata about the indices, aliases and data streams a request is going to touch. All other components of the security plugin rely on the index pattern resolution to reliably and efficiently provide this information.

However, as described in the following sections, the index pattern resolution currently operates in ways that are less than optimal.

Current situation

Performance

CPU profiling on the security plugin code shows that the core index pattern resolution code is one of the most CPU intense tasks of the request processing performed by the security plugin.

This is the code in question:

https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java

An analysis of the code shows that many things are done unnecessarily:

• The alias resolution is expensive because it is done by filtering the whole index set (which can be quite big); yet, the result of the alias resolution is used only in a single component which is often bypassed: the PrivilegeInterceptorImpl for implementing Dashboards multi tenancy.

• Data streams are broken apart into individual indices even if the privilege is granted on the data stream itself; as data streams can have many backing indices, this can be also expensive.

• Generally, lots of copying and filtering of potentially large data structures is going on

Structure

The class IndexResolverReplacer must have white-box knowledge of the requests it wants to process. This can be seen here:

https://github.com/opensearch-project/security/blob/19a41b8f01d00adbeec51b35788d38e18a2f2adc/src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java#L657-L836

This is a fragile way to implement things and requires a big maintenance effort. Newly actions added to the core must/shoule be reviewed whether they need special treatment here.

do_not_fail_on_forbidden

The class IndexResolverReplacer additionally implements the index replace algorithm required by the do_not_fail_on_forbidden mode. As the class does not implement it in a universal way, there are many issues with that mode as described in #3905.

New approach

Goal of this issue is to completely replace the IndexResolverReplacer implementation which:

• Leverages core interfaces to efficiently get the "true" resolved indices from a request
• Avoids unnecessary index resolutions
• Can be leveraged to implement the refined index authorization as described in #3905

Strategy

• Conceive core interfaces to be implemented by action requests to provide true index information
• Create prototype implementation (might also leverage caching of resolution result in request object)
• Review usage of IndexResolverReplacer in security plugin and conceive a new light weight replacement
• Implement refined index authorization (gated by feature flag)

Start with src/main/java/org/opensearch/security/resolver/IndexResolverReplacer.java, especially the request-specific handling around lines 657-836, and review the problems described in #3905. Map current usages of IndexResolverReplacer before evaluating the strategy checklist; done requires a reviewed replacement based on core request interfaces, with refined index authorization addressed behind a feature flag.