Security Information is removed from the TheadContext when plugins call on ThreadContext.stashContext

When plugins need to run actions outside of the currently authenticated user's context, they will call on ThreadContext.stashContext to assume a fresh context where they can run transport actions in a trusted manner. There are a few issues in the Security backlog that talk about this process in detail:

• https://github.com/opensearch-project/security/issues/2487
• https://github.com/opensearch-project/security/issues/2860

An example block looks like:

try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) {
   // Run code in this block with fresh context which allows a plugin to enter a trusted mode
   // PrivilegeEvaluation is skipped in this block as the SecurityFilter exits early here: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L315-L322
} catch (Exception e) {
    LOG.error(e);
    listener.onFailure(e);
}

As part of Centralized Scheduled Job Identity Management the security plugin would need access to the currently authenticated user even if the thread context has been stashed. There needs to be a mechanism (SecurityContext or something similar) where there is a carrier of Security information that is available even if the ThreadContext has been stashed. For Centralized Scheduled Job Identity Management, that would mean that Job Scheduler can listen in on index operations into plugins jobs indices and call on the security plugin to associate the newly created job to the currently authenticated user automatically. When the threadcontext is stashed, the security plugin currently has no idea who the currently authenticated user is. As a result, plugin developers have resorted to using roles injection where they will read the user from the threadcontext before stashing and save a copy of the user with the job. At job execution time they inject the roles back into the threadcontext. Ideally, plugin developers do not need to do that work as it is the responsibility of the identity/security system.