Sideproject: Devops - Automated network deployment

Background

Deployment tooling

We currently do not have the tooling to automate deployment of a new "joystream" network or even deployment of individual components to an existing network. The process is manual, intricate and time consuming.

The Development team is always relying on the "operations" team to run and deploy such networks and than leads to some delays to start testing new releases. During early release development before all new components are completed some basic local docker-compose environment are used by the development teams to test their changes.This is useful to some extent, and typically only near the end of a release where the various teams have completed features of their products, will a new test network be deployed.

With recent new use of M1 processor based laptops even local docker containers for joystream-node has become unusable.

The lack of automated deployment tools is reflected in the need to write length and detailed guides for the community on how to deploy validator nodes and various other services such as a storage node. This documentation needs to be updated often with every release and is also time consuming.

Integration testing

Integration tests are executed for new pull-requests in a github workflows with a very basic network setup (similar to what a local developer would test against) This is not sufficient and ideally the tests should be performed on a more realistic deployment.

Project Goal

The goal of this project is to develop the tooling for deploying new networks in a configurable and composable fashion for production deployment and testing purposes on both remote and local Kubernetes clusters.

Requirements

The tooling should facilitate:

• Single manifest file as network configuration with all required configuration parameters for network deployment
• Dynamically create and configure new cluster - geographically dispersed (platform independent, support AWS or Linode, or local cluster (linux/mac docker desktop))
• Host machines in cluster to compose multiple architectures if possible.
• Build docker images from source (potentially from multiple git repositories)
• Initialize network genesis state chainspec
• Deploy initial validator joystream-nodes
• Deploy additional nodes and turn into a validator
• Dispatch extrinsics to configure the network with additional state (workers, accounts, members etc)
• Handle domains and ssl certificates required by web services

Joystream network components

• 3 initial validators (joystream-node)
• 1 RPC (joystream-node)
• 1 Storage Node (joystream-node, colossus, ipfs node)
• 1 Query Node + (joystream-node, indexer and indexer gateway, processor, redis, postgedb, graphql server)
• 1 Pioneer instance (static website)
• 1 Atlas instance (static website)
• 1 Faucet (web service)

Start with .github/workflows/run-network-tests.yml and the listed validator and storage-provider deployment guides to understand the current network setup. Review the requirements for a manifest-driven deployment across local and remote Kubernetes clusters, including network components and genesis configuration. Done means the deployment tooling can create and configure the described network for production and testing.