Clarifying "experimental features" in OpenSearch [DISCUS][BUG]

What is the bug?

While we have a description of what an "experimental feature" feature can be used for in OpenSearch, we don't actually describe what the criteria need to be met to graduate from experimental to a general availability (GA).

Additionally, while we provide a general heuristic, we have plenty of features that are launched without ever being experimental, even those that could probably have benefited -- we leave the decision on both whether a feature should be marked as experimental and whether it's ready to graduate entirely up to the maintainers of the individual repos.

Finally, the OpenSearch Project does do security reviews for some features (you can see more information here). On the OpenSearch Project repos I've worked in, we (as maintainers) have considered a sign-off from AWS appsec following this process a requirement to graduate out of experimental for some features, but like much of the process around experimental features, that requirement not documented anywhere and is not universally adopted.

This has caused some confusion for contributors because the "rules of the road" have never been fully defined. So, I propose we define them :)

Here are a couple of questions to kick us off that are the top of my mind, but feel free to add your own:

0/ What purpose should experimental feature serve? Do we still agree with this description?

1/ Should the OpenSearch Project have rules about a) what features need an "experimental" phase and b) what a feature needs to graduate to GA at the project level, the repo level or something else?

Assuming we do want some kind of guidelines...

2/ Do we want criteria in place that, if your feature meets them, you must have an experimental phase?

3/ What criteria should we look at for graduation? Security? Scalability? Testability?

4/ Do we want to set a limit on how long a feature can be experimental?

5/ Who decided if those criteria have been met?

Please note: I'd like to focus this discussion on what (if any) criteria The OpenSearch Project wants to have around experimental/GA features. Downstream providers and products will have their own criteria, including AWS OpenSearch Service. We (OpenSearch Project) can certainly craft our criteria to make a GA launch on any particular service easier, but they should still be considered separate processes. Additionally, while a downstream project make make decisions based on us (e.g. "we will not expose any experimental feature to our customers"), we should not take a dependency based on them or their actions (e.g. "The OpenSearch Project will not mark a feature GA until XYZ Service agrees that it's GA").

WDYT?

Start by reviewing the experimental-features section of CONTRIBUTING.md and the security-reviews section of RELEASING.md, then read the issue discussion. Done would require an agreed project-level policy defining experimental-feature use, GA graduation criteria, security expectations, time limits, and decision ownership, followed by updates to the relevant documentation.