Proposal: Experimental features interface / API

This issue is a proposal to implement for a product-wide approach for experimental features in Gitpod.

Context / Motivations

We have the concept of "VS Code Insiders", an experimental pre-release designed for power-users to be able to test out features in VS Code before they're rolled out to the full user base (see below) [1].

But the IDE interface is now starting to get cluttered, and the distinction between pre-release versions isn't very clear (it already was not entirely clear what the insider's version is).

We are now considering applying the same approach (insiders/early access IDE's) for EAP [1] versions of JetBrains IDE's, where we have a pre-release version and a stable.

Additionally, we've had challenges in the past of users not knowing when a feature is a beta / pre-release vs a stable one (e.g. https://github.com/gitpod-io/gitpod/pull/8783). This causes potential to upset or confusion for users and causes additional support burden due to increased GitHub tickets, questions from users, etc. The docs labels help somewhat, but do not guarantee that a user understands fully the implications of an experimental feature:

We have beta labels in-app, but again, do not ensure that a user is entirely aware of implications of an experimental feature:

The same challenge applies also for experimental commands within gp cli

Solution

• A clear description of the experimental feature (to the user) e.g. what the feature does, what are the impacts
• A way to toggle on/off the feature

From a technical design perspective, the solution needs to be able to be easily plugged into other aspects of the application, so a logical API and/or interface is essential. The solution should also play nicely with any existing feature flag behaviours, and potentially could be used for A/B testing in the future. We'd also need to consider how this might look when integrated with other API's such as gp CLI.

Related: https://github.com/gitpod-io/gitpod/issues/6826#issuecomment-976238323
Related internal Slack discussion.

CC: @jakobhero

The issue names no files, tests, or implementation entry points. Start by reviewing Gitpod's existing feature-flag behavior and the gp CLI integration, then compare how VS Code Insiders and JetBrains EAP features are exposed. Done would require an agreed product-wide API/interface, user-facing descriptions and toggles, and a defined integration path for experimental CLI commands.