Installation script refinement

After installing OpenPAI through the quick start script for many times, and addressing some deployment-related issues, I find there are some general problems in current installation and management process.

Some main issues include:

Duplicate configuration file #5151

In quick start script, the user provides worker.csv, master.csv and config.yml. The script then generates hosts.yml, openpai.yml, layout.yaml, services-configuration.yaml based on the three files.

During adding/removing nodes, the user should modify hosts.yml, layout.yaml and services-configuration.yaml.

During cluster management, the user uses layout.yaml and services-configuration.yaml.

During uninstallation, the user should use hosts.yml and openpai.yml.

It's hard to manage/understand a lot of kinds of configuration.

Auto-generated hived config causes trouble

Currently, the hived config is generated using a script. The script will find the minimal resource among all workers, and use the minimal resource as the VC resource. However, it is a common case that some machines are broken in a big cluster. A broken machine will cause the config to be wrong entirely.

Lack environment cleaning script during installation

Failure in installation is common. Our script and kubespray script is supposed to be idempotent. So the user can fix the issue and try again if any error encountered. But the idempotence is hard to maintain. We can provide the user with a environment cleaning script so he can get rid of some issues quickly.

Lack support for installing CPU-only workers from scratch

Many customers raise issues: #5084 #4945 #4281

Lack support for Singlebox installation

Many customers raise issues: #4967 #4614 #4566 #4281

Dev-box machine is always required

Dev-box machine is recommended but I think it should not be required, especially for those who want singlebox installation.

Log

Current installation log is very verbose. A lot of output lines are from kubespray. Users have no idea of the installation progress, and may be confused by the reported error.

Other issues

Some other issues are listed in #5001 . Also, #5001 and this issue have some overlap.

---

To address the above issues, my basic idea is to use layout.yaml and config.yaml during installation. Users only need to know layout.yaml, config.yaml, and services-configuration.yaml. In layout.yaml, we can address issues like CPU-only installation, single-box installation, and auto-generated hived config problem.

1. During installation, user provides layout.yaml and config.yaml

layout.yaml for normal GPU worker installation:

machine-sku:
  master-machine:
    model: cpu-node
    mem: 60GB
    cpu:
      vcore: 24

  gpu-machine:
   model: nvidia-gpu-node
    mem: 220GB
    cpu:
      vcore: 24
    gpu:
      type: K80
      count: 4

machine-list:
    - hostname: pai-master
      hostip: 10.0.0.1
      machine-type: master-machine
      pai-master: "true"
    - hostname: pai-worker1
      hostip: 10.0.0.2
      machine-type: gpu-machine
      pai-worker: "true"
    - hostname: pai-worker2
      hostip: 10.0.0.3
      machine-type: gpu-machine
      pai-worker: "true"

layout.yaml for pure-CPU worker installation:

machine-sku:
  master-machine:
    model: cpu-node
    mem: 60GB
    cpu:
      vcore: 24

  cpu-machine:
    mem: 220GB
    cpu:
      vcore: 24

machine-list:
    - hostname: pai-master
      hostip: 10.0.0.1
      machine-type: master-machine
      pai-master: "true"
    - hostname: pai-worker1
      hostip: 10.0.0.2
      machine-type: cpu-machine
      pai-worker: "true"
    - hostname: pai-worker2
      hostip: 10.0.0.3
      machine-type: cpu-machine
      pai-worker: "true"

layout.yaml for single-box GPU installation:

machine-sku:
  gpu-machine:
    model: nvidia-gpu-node
    mem: 220GB
    cpu:
      vcore: 24
    gpu:
      type: K80
      count: 4

machine-list:
    - hostname: pai-master
      hostip: 10.0.0.1
      machine-type: gpu-machine
      pai-master: "true"
      pai-worker: "true"

These are checked during installation:

• If the resources are set properly, and if the resource requirement is met.
• There should be pai-master and pai-worker.
• Only one pai-master.

model can be mapped to a group of config validating, requirement checking, and hived scheduler config generating script.

config.yaml includes some customized settings during installation:

user: <ssh-password>
password: <ssh-password>
version: v1.3.0, v1.4.0, ..., etc.

# Customized branch and tag. Cannot be set together with version.
# Can be used in debug.
# customized_branch_name: master
# customized_docker_image_tag: int

#############################################
# Ansible-playbooks' inventory hosts' vars. #
#############################################
# ssh_key_file_path: /path/to/you/key/file

#####################################
# OpenPAI's service image registry. #
#####################################
# docker_registry_domain: docker.io
# docker_registry_namespace: openpai
# docker_registry_username: exampleuser
# docker_registry_password: examplepasswd

################################################################
# OpenPAI's daemon qos config.                                 #
# By default, the QoS class for PAI daemon is BestEffort.      #
# If you want to promote QoS class to Burstable or Guaranteed, #
# you should set the value to true.                            #
################################################################
# qos-switch: "false"

########################################################################################
# Advanced docker configuration. If you are not familiar with them, don't change them. #
########################################################################################
# docker_data_root: /mnt/docker
# docker_config_file_path: /etc/docker/daemon.json
# docker_iptables_enabled: false

## An obvious use case is allowing insecure-registry access to self hosted registries.
## Can be ipaddress and domain_name.
## example define 172.19.16.11 or mirror.registry.io
# openpai_docker_insecure_registries:
#   - mirror.registry.io
#   - 172.19.16.11

## Add other registry,example China registry mirror.
# openpai_docker_registry_mirrors:
#   - https://registry.docker-cn.com
#   - https://mirror.aliyuncs.com

#######################################################################
#                       kubespray setting                             #
#######################################################################

# If you couldn't access to gcr.io or docker.io, please configure it.
# gcr_image_repo: "gcr.io"
# kube_image_repo: "gcr.io/google-containers"
# quay_image_repo: "quay.io"
# docker_image_repo: "docker.io"
# etcd_image_repo: "quay.io/coreos/etcd"
# pod_infra_image_repo: "gcr.io/google_containers/pause-{{ image_arch }}"
# kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
# hyperkube_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/hyperkube"

# openpai_kube_network_plugin: calico

# openpai_kubespray_extra_var:
#   kay: value
#   key: value

#######################################################################
#                     host daemon port setting                        #
#######################################################################
# host_daemon_port_start: 40000
# host_daemon_port_end: 65535

2. Requirement check and hived config generation based on layout.yaml

Previous hived config is generated after k8s is deployed. Now, we can first examine the resource specified by users, and then generate hived config for them.

We can put homogenous skus in one VC.

3. Provide an environment cleaning script

Use kubespray's reset.yml is enough.

4. Add and remove node using layout.yaml

Use layout.yaml generates hosts.yml and run kubespray script.

5. Uninstallation using layout.yaml and config.yaml

6. Refine logs

Show steps and progress.