[BUG] java.lang.NoSuchMethodError in a REST handler causes a OpenSearchSecurityException

What is the bug?

I installed an incompatible plugin into OpenSearch that has a REST handler. That handler raises a NoMethodError in this particular instance of OpenSearch, causing the following API response.

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "Unexpected exception api"
      }
    ],
    "type": "security_exception",
    "reason": "Unexpected exception api"
  },
  "status": 500
}

The root cause is a NoSuchMethodError in the plugin.

[2024-01-11T19:48:17,964][ERROR][o.o.s.f.SecurityFilter   ] [b6e284bbf0cb] Unexpected exception java.lang.NoSuchMethodError: 'org.opensearch.core.xcontent.XContentBuilder org.opensearch.rest.RestController.toXContent(org.opensearch.core.xcontent.XContentBuilder, org.opensearch.core.xcontent.ToXContent$Params)'
java.lang.NoSuchMethodError: 'org.opensearch.core.xcontent.XContentBuilder org.opensearch.rest.RestController.toXContent(org.opensearch.core.xcontent.XContentBuilder, org.opensearch.core.xcontent.ToXContent$Params)'
	at org.opensearch.plugin.api.action.APIResponse.toXContent(APIResponse.java:74) ~[?:?]
	at org.opensearch.plugin.api.RestAPIAction$1.buildResponse(RestAPIAction.java:72) ~[?:?]
	at org.opensearch.plugin.api.RestAPIAction$1.buildResponse(RestAPIAction.java:68) ~[?:?]
	at org.opensearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:53) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.plugin.api.action.TransportAPIAction.doExecute(TransportAPIAction.java:30) ~[?:?]
	at org.opensearch.plugin.api.action.TransportAPIAction.doExecute(TransportAPIAction.java:21) ~[?:?]

But this causes

[2024-01-11T20:09:49,425][WARN ][r.suppressed             ] [a0fd2fdfc7e0] path: /_plugins/api, params: {}
org.opensearch.OpenSearchSecurityException: Unexpected exception api
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:479) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.plugin.api.RestAPIAction.lambda$prepareRequest$0(RestAPIAction.java:68) [api-2.11.1.jar:2.11.1]
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.security.filter.SecurityRestFilter.lambda$wrap$2(SecurityRestFilter.java:187) [opensearch-security-2.11.1.0.jar:2.11.1.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:323) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:414) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:252) [opensearch-2.11.1.jar:2.11.1]

Reproduce

1. OpenSearch 2.11 standard docker.
2. Build and install opensearch-api from https://github.com/dblock/OpenSearch/tree/2.11.1-api-plugin with ./gradlew :plugins:api:assemble.
3. curl -u admin:admin https://localhost:9200/_plugins/api

What is the expected behavior?

An internal server error, not a security_exception.

Do you have any additional context?

Installing a plugin built from https://github.com/dblock/OpenSearch/tree/2.11.1-api-plugin.

Reproduce the failure with the OpenSearch 2.11 Docker image, the api plugin build, and the curl request described in the issue. Start in SecurityFilter.java around apply0 and apply, then trace how the REST handler exception becomes an OpenSearchSecurityException. Done means the same plugin failure returns an internal server error rather than a security_exception.