Cluster Infrastructure

FIXME Link here to doc about our current Kubernetes cluster and hosting setup.

• Monitoring
  • Add customized dashboards using Grafonnet to kube-prometheus.
    • cert-manager
    • openebs
  • Configure Alerts and send notifications to a matrix-channel.
    • Maybe: matrix-alertmanager
  • Add website analytics with Fathom.
  • Create public status page with overview of current apps.
  • Regularly check observatory.mozilla.org for all public sites.
• Authentication
  • OpenID Connect via Keycloak for kube-apiserver and apps.
  • Add gangway.
• Security
  • Create restricted Pod Security Policy to only allow non-root.
  • Default deny all ingress traffic
  • RBAC
• Shared services
  • Kinto
  • Postgres
  • Minio
  • Elasticsearch
• Backup
  • Push database snapshots and filestores regularly so some s3 storage.
• Stability
  • Automatically replace the oldest node every twelve hours with a fresh one. Maybe with the help of kured.
  • Make sure limits are set with every pod.
  • Make every service be backed by at least two replicas. Label apps that can't deal with this.
  • Set PodDisruptionBudget for all apps.
  • Set recommended labels for all resources.

Random Ideas

• Try varnish with traffics and crashes.
• Add blackbox exporter for our public services.

Start with the checklist in this issue and its linked Kubernetes, Grafonnet, kube-prometheus, and Matrix Alertmanager references; no repository file or test is named. Narrow the work to one unchecked infrastructure item before changing configuration. Done means the selected item is implemented and its operational outcome is verified.