Add ps-operator training class (16-hour Percona Operator for MySQL/PS course)

Summary

Add a new ps-operator training class — a 16-hour, hands-on course on the Percona Operator for MySQL based on Percona Server (PS Operator). Modeled on the existing Mongo two-deck Tutorial / Part II format.

Today the repo has no class slug that delivers a PS-Operator-flavored curriculum: the existing mysql-k8s slug provisions only a generic Minikube playground via node1, and the bare-metal pxc slug covers a different product (PXC, with synchronous Galera replication). The PS Operator is architecturally distinct — async replication + Orchestrator, or Group Replication with HAProxy/Router; no ProxySQL; new CRD kinds (PerconaServerMySQL, ps.percona.com); operator-managed PITR via a dedicated binlog server — and warrants its own class.

Infrastructure

Per-team layout: one EC2 jumphost (new psop machine type, t3.2xlarge) running a single-node kind cluster locally. Students SSH to the jumphost as rocky; kubectl is pre-installed and on PATH (including /usr/bin for sudo-invoked tools). No second cloud resource per team — kind is fast to provision, free, and lets Lab 1 ("install the operator") actually mean something because the student is installing CRDs into their cluster.

Why kind on jumphost vs alternatives (decided during design):

• EKS, one cluster per team — adds ~$500/class in control-plane + node cost, ~15 min/cluster to provision. Defer; can be added as an alternate slug later.
• Shared EKS with namespace-per-team — rejected; CRD install is cluster-scoped so Lab 1 only works once.

In-cluster S3 backend: Garage (not MinIO)

PITR labs need an S3-compatible backend. We deploy Garage v2.3.0 in-cluster (single Pod, single PVC, single Service) on each team's kind cluster. v2.3.0 is the first release with the env-var-driven default-key/bucket bootstrap, so the setup is one Deployment with no separate bootstrap Job (though we keep an explicit garage bucket create / key import / bucket allow step in ansible for determinism across PVC reuse).

MinIO was the obvious first choice but is unmaintainable as of 2026:

• Feb 2025: admin UI features stripped from Community Edition (commercial-only).
• Dec 2025: project declared maintenance mode.
• Feb 2026: GitHub repository archived. No security patches, no community binaries.

Useful pedagogical side effect — backups in the PS Operator are provider-agnostic; the same CR works against AWS S3, Backblaze B2, Wasabi, or any other S3-compatible endpoint. The MinIO situation becomes one slide.

Pinned versions

	Version	Released
PS Operator	v1.1.0	2026-04-17
Garage	v2.3.0	2026-04-16
kind	v0.27.0	—
kubectl	v1.31.0	—
helm	v3.16.4	—
k9s	v0.32.7	—

All in roles/ps-operator/defaults/main.yml for one-line bumps.

Lab outline

Two decks for the 16-hour format, slide source lives in percona/training-material (separate PR).

Deck A — "Kubernetes PS Operator Tutorial" (Day 1, ~85 slides)

1. Containers / Docker / Orchestration / Kubernetes overview (reused from existing decks)
2. PS Operator overview — replication-topology choice, when to use PS vs PXC operator
3. Lab 1 — Configure and Deploy (incl. anti-affinity edit + CRD wait)
4. Lab 2 — Backups and Recovery (on-demand + scheduled to Garage S3, restore)
5. Lab 3 — Modify mysqld parameters via spec.mysql.configuration
6. Lab 4 — Scaling (horizontal hands-on; vertical and PVC expansion as spec-only on kind)
7. Lab 5 — GR-native failover (kill primary, watch GR elect new one)

Deck B — "PS Operator – Part II" (Day 2, ~80 slides) — async + Orchestrator deep-dive, cluster mgmt (pause/SmartUpdate/replication), PMM, migration paths, troubleshooting, best practices.

Lab 1 deliberately leaves the operator install to the student — ansible only stages the cloned repo, not the applied resources.

Validation

Dry-run completed end-to-end on AWS us-west-2 across 4 full provision cycles:

• make setup class=ps-operator client=TEST teams=1 region=us-west-2 — clean 32-task ansible run (~6 minutes including AWS provisioning and SSH-ready gate)
• All 5 Day-1 labs hands-on verified: deploy, backup/restore with Garage S3 (checksum match), config tune, 1→3 scale, GR-native failover with primary promotion in seconds
• Day-2 testable bits: pt-k8s-debug-collector, pause/resume, status conditions, clean CR + bundle uninstall
• make teardown cycles cleanly between runs

Bugs found and fixed during dry-runs (full details in the commits):

• setup-class.sh AMI auto-detect was broken in two ways: hardcoded -p dummy required a non-existent dummy VPC, and head -n 1 picked the oldest AMI (printAmis() sorts ascending by date).
• setup-class.sh had no SSH-ready gate; ansible's Gathering Facts task timed out connecting to a still-booting EC2.
• start-instances.php GETANSIBLEHOSTS matched terminated instances (Name tag persists for an hour after teardown), producing duplicate inventory entries with empty IPs after a teardown/redeploy.
• percona-toolkit was installing 3.6.0 from the tools repo (enabled as side-effect of ps-84-lts); now explicitly disabled so pt channel's 3.7.1 wins.
• pt-k8s-debug-collector lives at https://percona.com/get/ as a Go binary, not in the percona-toolkit source tree at any tag.
• Rocky 9's sudo secure_path excludes /usr/local/bin; ansible shell tasks invoking kind/kubectl now use absolute paths, and /usr/bin/kubectl is symlinked so pt-k8s-debug-collector and similar sudo-invoked tools find it.
• Garage v2.3.0 quirks: image has only CMD ["/garage","server"] (no ENTRYPOINT), rpc_bind_addr is required even single-node, and GARAGE_DEFAULT_* env-var auto-bootstrap only fires on truly-empty PVCs — added an explicit garage bucket create / key import / bucket allow step that's deterministic and idempotent.

Cost of the validation: ~$2 in us-west-2 (4 provision cycles × ~30 min × t3.2xlarge).

Scope of this PR

Infrastructure side only: new setup-class.sh slug, new psop machine type, new roles/ps-operator/ ansible role, README updates. Slide content lives in percona/training-material and is a separate PR.

Start with setup-class.sh, the psop machine-type configuration, and roles/ps-operator/defaults/main.yml; inspect how existing class slugs and Ansible roles are wired in. Run make setup class=ps-operator client=TEST teams=1 region=us-west-2, then verify the documented Day 1 validation and clean teardown. Done means the infrastructure provisions and the README reflects the new class.