FlowFuse / FlowFuse/flowfuse

Support deploying instances to remote k8s clusters

Open
#7,305 2 comments 1 reaction 0 assignees View on GitHub
customer request epic
Dominant language
JavaScript
Stars
400
Forks
89
Avg merge
1d 20h
Merged PRs (30d)
149

Description

### Description

Our current architecture only supports running hosted instances within the 'local' k8s cluster. This epic covers the technical work to add support for managing hosted instances on non-local k8s clusters.

> [!note]
> There's a terminology clash here when talking about local and 'remote' - as 'remote' has meaning in the FF platform where an instance is run via the Device Agent.
> To try to minimise confusion, I'll use 'satellite cluster' to refer to the non-local/remote k8s cluster.

## Current Architecture

The current platform architecture uses [driver-k8s](https://github.com/flowfuse/driver-k8s) to provide the interface between the forge app and the underlying k8s api.

When the forge app wants to create a new instance, it passes the request the to the driver. The driver does the work to create the various k8s artefacts needed for the instance by calling the k8s apis directly.

The `nr-launcher` is the entry-point for the instance. It communicates back to the forge app, but also exposes an HTTP api the driver to use to manage the instance.

## Proposed Architecture

There is currently a lot of code in the driver to create all of the individual artefacts needed to the run instance. We will migrate this code to a k8s operator. The driver will then be responsible for assembling the complete yaml needed and pass it to the operator to do the work.

This operator can then be installed into the satellite clusters to provide the same functionality.

We need to verify the operator will provide all of the lifecycle operations we require.

## Communication with the satellite cluster

The driver currently communicates with the individual nr-launchers privately within the cluster. We won't want satellite clusters to expose the nr-launcher admin api externally. Nor do we want the k8s api to be exposed.

We will create an application that runs in the satellite cluster that creates a secure MQTT connection to the broker in the primary cluster. This will allow command/control messages to be sent to the satellite cluster securely.

## Setting up a satellite cluster

1. Admin user registers the satellite cluster details in the forge app.
2. Access to satellite cluster can be scoped to specific teams.
3. When a user creates a new instance, if there are multiple clusters available to the team, the user can pick which cluster to deploy the instance to.

## High level task breakdown

- Create K8S Operator
- Update driver-k8s to use the operator or sends message to satellite agent
- Create satellite agent (inc helm chart)
- Admin UI for managing satellite agents
- Create Instance UI updates to allow satellite cluster selection

### Which customers would this be available to

Enterprise Tier Only (EE)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.