Epic: GPU Support on SaaS

Callout: Hey 👋 Are you interested in this feature? I would be interested in talking to you: https://calendly.com/andre-gitpod/15-minute-product-feedback

We want to fully support GPUs to enable different workloads, such as data science, particularly on self-hosted installations.

Open questions:

• Would we be able to dynamically assign GPUs to workspaces? Not right now
• Would we be able to share one GPU between multiple workspaces? Preferably not right away, given performance and security concerns [video]
• Would we be able to support all types of GPUs or would need to select a subset? What subset? Given the focus on SaaS, we will just pick one and go with that.
• Are we able to confidently release GPU support for self-hosted without launching it also on SaaS? No
• Is there a limitation on what and how many GPUs you can attach to a given instance type? We will need to research, maybe there are. Depending on that we might want one workspace per node, or multiple workspaces per node with multiple GPUs.

Options:

• Just another class — We don't share GPUs across workspaces. We have a separate node pool / workspace class that has a defined workspace density, which equals the number of GPUs. We would launch on SaaS first, as we need to

Concerns:

• Workspace classes are hard-coded in server. Probably resolved in #8261.
• The user will have to keep on changing their user settings to use GPUs or not. We should have workspace class project setting. And in the future, it might make sense to allow Data Scientist to select whenever they open the workspace or inside the workspace.

Dependencies:

• (for self-hosted) Self-hosted will need to be able to configure multiple workspace classes.
• (Preferably) Per project workspace class setting.
• (Must) Possibility of scaling a node pool to zero.

Things we know we will have to do:

• https://github.com/gitpod-io/gitpod/issues/8396
• To mount the GPU device in the workspace. Probably replicate the work we did for the fuse device.

Related request: https://github.com/gitpod-io/customers/issues/24

Start by reviewing issue #8396 and the existing fuse-device work referenced in the issue. Then investigate workspace classes, multiple self-hosted workspace classes, project workspace settings, and node-pool scaling constraints. Done would require a defined GPU-support design covering SaaS and self-hosted installations, including GPU device mounting and the listed dependencies.