RFC: Volume snapshot based backups and replica provisioning

Summary

Add an opt-in interface to vitess-operator that takes CSI VolumeSnapshots of tablet
data volumes as a backup method, and uses those snapshots to provision new replica
tablets (PVC dataSourceRef). This was proposed for Vitess core in
vitessio/vitess#6952 and closed with the conclusion that the operator is the right
place for it:

• This doesn't belong in core vitess because it is unclear how you would restore
• Operator has the correct scope to implement a volume snapshot backup/restore

— @deepthi, vitessio/vitess#6952 (2021-05-24)

As far as I can tell nobody has picked this up on the operator side since. I'd like
to implement it in phases, and want to agree on the design first. CloudNativePG's
spec.backup.volumeSnapshot interface is used as a reference point throughout, with
the places where MySQL/Vitess semantics differ called out explicitly.

Motivation

• Speed: file-based backups scale with data size; snapshots are near-constant time.
  The original proposal reported ~3h for a 1.3TB shard with the builtin engine vs
  ~3min with a GCP disk snapshot.
• Cost: cloud snapshots are incremental at the block layer.
• Replica provisioning: restoring a new replica from object storage is the slow path
  for scale-up and node replacement today. Provisioning the PVC from a recent
  snapshot and letting GTID auto-positioning catch up is much faster for large shards.

I ran a proof-of-concept of the full flow against operator v2.17.0 / Vitess v24.0.2
on kind (laptop scale, so no performance claims — the numbers above stay attributed
to #6952). What the PoC verified end to end: snapshot → new PVC via dataSourceRef
→ mysqld comes up (crash recovery included for a SIGKILL-under-load image, with
gtid_executed reconstructed to the exact crash point) → GTID auto-positioning
catch-up (a 1,359s replication gap closed in 7.5s) → CHECKSUM TABLE matches the
primary. The PoC also surfaced two design constraints that are folded into the
sections below (mysqld supervision vs. cold fencing, and tablet-UID coupling of the
data volume layout).

Goals (initial scope)

1. Scheduled, per-shard volume snapshot backups, orchestrated by the operator,
   with a cold (default) and an opt-in online mode.
2. Provisioning new replica tablets from the most recent usable snapshot
   (PVC dataSourceRef), with GTID auto-positioning for catch-up.
3. No changes to Vitess core. The whole flow is composed from existing primitives
   (ChangeTabletType, StopReplication/StartReplication, mysqlctld shutdown,
   Kubernetes VolumeSnapshot API).

Non-goals (initial scope — listed so they're explicit, not forgotten)

• Full-shard disaster recovery from a snapshot (primary lost). Restore semantics
  there involve reparenting and RPO questions that deserve their own phase.
• Bootstrapping a new cluster/keyspace from snapshots (e.g. staging clones).
• Point-in-time recovery. Vitess incremental backups are anchored to manifests in
  BackupStorage and cannot currently layer on top of a snapshot base.
• Integration with vtctldclient GetBackups / BackupStorage. Snapshot backups live
  on a different axis (Kubernetes objects, not BackupStorage files) and the initial
  scope keeps that split explicit rather than papering over it.
• Multi-volume tablets. If the data dir, redo log and binlogs are not on a single
  atomically-snapshotted volume, the consistency argument below does not hold.
  Until CSI VolumeGroupSnapshots are broadly available, the operator should refuse
  to snapshot such layouts.

Consistency model

This is the part that made restore "unclear" for core, so spelling it out:

A CSI volume snapshot is point-in-time atomic at the block layer — equivalent to a
power-loss image. What makes that image usable is a three-layer argument:

1. Engine: InnoDB crash recovery brings the image to a transactionally
   consistent state as of the snapshot instant (redo roll-forward, undo rollback).
2. Replication position: with binlogs enabled, MySQL reconstructs
   gtid_executed from the binlog during recovery, and binlog↔engine consistency
   is maintained by the internal two-phase commit (binlog XA recovery: prepared
   transactions missing from the binlog are rolled back, the binlog tail is
   trimmed). The restored instance therefore knows exactly which GTIDs it has.
3. Catch-up: GTID auto-positioning re-joins the shard from that position.
   Stale relay logs on the image are irrelevant under auto-positioning.

For the online (hot) mode, this argument holds only under preconditions:

• sync_binlog=1 and innodb_flush_log_at_trx_commit=1 on the target. With
  relaxed settings, transactions committed in the engine can be missing from the
  binlog on a crash image; the reconstructed GTID set then has holes, the source
  re-sends those transactions, and they get applied twice — silent drift.
• Data dir, redo and binlogs on one volume (single snapshot atomicity domain).

Because the failure mode of a violated precondition is silent replication drift,
online mode is opt-in (online: false by default), and the operator probes the
preconditions before allowing it. This mirrors CloudNativePG's online flag but
with the default flipped: CNPG needs engine cooperation (pg_backup_start) mainly
because it snapshots PGDATA and WAL as separate volumes; a single-volume MySQL
tablet needs no engine cooperation, but inherits MySQL crash-safety settings as a
hard precondition instead.

The cold mode (default) sidesteps all of this by taking the snapshot from a
cleanly shut down mysqld, so it is safe regardless of durability settings.

In both modes, a restored replica can only catch up if the source still has the
binlogs covering the snapshot age. The operator cannot verify binlog retention on
its own; this is documented, and a replica that cannot catch up surfaces as a
failed catch-up event with a fallback to the normal provisioning path.

Design

API sketch

# VitessCluster
spec:
  backup:
    volumeSnapshot:
      className: <VolumeSnapshotClass name>
      online: false            # default: cold. true requires precondition probe to pass
      target:
        tabletType: replica    # replica (fence in place) | rdonly (dedicated pool)
      retention:
        maxCount: 7

VitessBackupSchedule gets a third strategy method, volumeSnapshot, alongside
the existing vtbackup-pod and vtctldclient-job methods.

Restore side, on a tablet pool:

dataSource:
  volumeSnapshot: latest   # inject dataSourceRef into new tablet PVCs

Cold backup flow

1. Pick a target: never the primary; shard must be healthy; require at least two
   replicas unless explicitly overridden (single-tablet dev shards); refuse if a
   previous snapshot operation still holds a fence on this shard (same contract as
   CNPG refusing cold backups while instances are fenced).
2. ChangeTabletType → DRAINED, StopReplication, record gtid_executed.
3. Clean mysqld shutdown via mysqlctld. The fence window is dominated by the
   buffer pool flush on shutdown, not by the snapshot itself.
4. Create the VolumeSnapshot. Resume as soon as the snapshot is cut — not
   readyToUse, which on some drivers includes a background upload. The exact
   cut-vs-ready contract is driver-specific and needs to be stated in docs.
5. Restart mysqld, StartReplication, restore the tablet type. Catch-up is
   automatic via GTID auto-positioning.
6. Label/annotate the snapshot: cluster, keyspace, shard, GTID set, MySQL version,
   timestamp.

Failure safety. The number one failure mode is a leaked fence. Every error path
attempts an unconditional unfence, and a reconciler acts as a safety net for
tablets left in DRAINED by a dead snapshot job.

Supervision must be part of the fence (PoC finding). In a stock operator pod,
shutting mysqld down cleanly does not keep it down: the in-pod management layer
restarted it in place about 8 seconds later (observed as a Mysqld.Start served by
mysqlctld; no container restart). A naive shutdown-then-snapshot loses that race —
in my PoC the snapshot cut landed after the restart, and the run was only saved by
skip_replica_start freezing the GTID state. The cold flow therefore needs an
explicit way to hold mysqld down for the duration of the cut — a vttablet/mysqlctld
primitive to suspend mysqld management, or a pod-level stop. This is the main
implementation question of the cold path (see open questions).

Online backup flow (opt-in)

Same as cold minus steps 2–3 and 5: probe preconditions (sync_binlog,
innodb_flush_log_at_trx_commit, single-volume layout), snapshot a serving
replica, annotate with a GTID lower bound (the exact position is discovered by
crash recovery at restore time). The probe is check-then-act, so a settings change
after the probe is not caught — documented limitation.

Restore flow (replica provisioning)

When the operator creates a PVC for a new tablet in an opted-in pool (scale-up or
replacement), it injects a dataSourceRef pointing at the newest ready snapshot
for that shard, subject to a MySQL version match from the snapshot annotations.
The tablet starts on the pre-populated volume, runs crash recovery if the snapshot
was online, and catches up via auto-positioning. If no usable snapshot exists, fall
back to the normal provisioning path and emit an event.

Two details verified/surfaced by the PoC:

• The non-empty data dir path is already handled: vttablet with
  --restore-from-backup logs "Attempting to restore, but mysqld already contains
  data. Assuming vttablet was just restarted." (v24 tabletmanager/restore.go) and
  proceeds as a restart. So no vttablet change is needed for restore to be skipped.
• The volume layout is coupled to the tablet UID: the data dir is
  vt_<tabletUID>/ and the generated my.cnf hardcodes those paths plus server-id.
  Restoring a snapshot into a new tablet UID (scale-up) therefore needs a small
  preparation step — rename the directory, regenerate my.cnf, and remove auto.cnf
  so a fresh server_uuid is generated (GTID state lives in
  mysql.gtid_executed/binlogs, so this loses nothing). Same-UID replacement needs
  none of this. An init container in the tablet pod is probably the right home.

Inventory

Initial scope: snapshots are plain VolumeSnapshot objects with a documented
label schema (kubectl get volumesnapshots -l planetscale.com/...). Whether they
should also be reflected as VitessBackup objects is left as an open question —
they would show up in kubectl get vitessbackups but never in
vtctldclient GetBackups, and I'd rather keep that split visible than blur it.

Phased implementation

1. PVC dataSourceRef plumbing for tablet pool volume claims. This revives the
   idea from #225, which was closed unmerged in 2021.
2. Snapshot choreography + schedule method: the cold/online flows above, the
   fence-leak reconciler, retention GC.
3. Restore injection + e2e + docs: pool-level dataSource, endtoend coverage,
   user docs including the driver cut-vs-ready caveat and binlog retention note.

Each phase is independently reviewable and useful (phase 1 alone already lets
users hand-provision from snapshots).

Testing note

Upstream e2e for this feature cannot use csi-hostpath for the online path: that
driver copies files (tar) rather than cutting at the block layer, and it fails
outright on a volume with a running mysqld (tar: file changed as we read it,
observed). e2e on kind would need a block-level test driver (e.g. topolvm), or
restrict itself to the cold path with mysqld fully stopped.

Open questions

1. Should snapshots be reflected into VitessBackup CRs, or stay label-only?
2. Snapshot ownership/lifecycle: owned by the VitessCluster (deleted with it) or
   independent? CNPG exposes this as snapshotOwnerReference: none|backup|cluster.
3. What is the right primitive to hold mysqld down during a cold cut? DRAINED +
   StopReplication does not prevent the in-pod management layer from restarting
   mysqld (observed, see the cold flow section). Options I can see: a
   tabletmanager RPC to suspend mysqld management, a mysqlctld-level hold, or a
   pod-level stop orchestrated by the operator.
4. Resume-on-cut vs resume-on-ready: acceptable to make this a documented,
   driver-dependent behavior, or should the operator wait for readyToUse
   conservatively (much longer fence windows on some drivers)?
5. Naming: volumeSnapshot as a VitessBackupSchedule method vs a separate
   controller/CRD.

References

• vitessio/vitess#6952 — original proposal and the decision that this belongs in
  the operator
• planetscale/vitess-operator#225 — earlier attempt at PVC data sources
• CloudNativePG volume snapshot backups:
  https://cloudnative-pg.io/docs/devel/appendixes/backup_volumesnapshot
• #811 — small manifest fix for a namespace footgun I hit while setting up the PoC

Start by reviewing the prior PVC dataSourceRef work in #225 and tabletmanager/restore.go, then trace tablet pool PVC creation and VitessBackupSchedule strategies. The phased scope covers PVC plumbing, snapshot choreography and restore injection, with e2e coverage and documentation; the open design questions need resolution before implementation can be considered done.