aws-samples / aws-samples/appmod-blueprints
[DR] Backup & restore capability for stateful workloads on spoke clusters
- Dominant language
- Shell
- Stars
- 105
- Forks
- 62
- Avg merge
- 11h 17m
- Merged PRs (30d)
- 76
Description
## Context
The platform currently provisions stateful workloads on spoke EKS clusters but offers no built-in backup/restore mechanism. Operators must bring their own tooling, with no GitOps-driven contract for protection policies. This issue captures the **functional requirements** for adding a backup & restore capability to `appmod-blueprints` spoke clusters. The technical solution (AWS Backup for EKS, Velero, Kasten, or other) is **not yet defined** and will be evaluated separately.
## Functional requirements
1. **Tiered protection policies** declared as code (e.g. gold / silver / bronze) with distinct RPO, retention, and cross-region semantics. Tenants opt into a tier — they don't compose individual backup parameters.
2. **GitOps contract**: tenants enroll a workload via a single declarative manifest (CR / annotation / label — TBD by design). No out-of-band AWS console steps, no platform-team ticket per tenant.
3. **Cross-region disaster recovery**: at least one tier MUST replicate backups to a secondary AWS region.
4. **Restore workflows** documented and testable for:
- in-place restore (same cluster, same namespace)
- cross-region restore (DR cluster in secondary region)
- clone-to-new-environment (stage-to-prod, blue/green, ephemeral envs)
5. **Coverage scope explicit** — what is protected vs what stays out:
- PVC data (crash-consistent at minimum, app-consistent where supported)
- Kubernetes manifests not derivable from Git (e.g. dynamic CRs)
- RBAC, ConfigMaps, Secrets — explicit decision per category
- What is **explicitly out**: secrets re-hydrated by External Secrets Operator, anything Argo CD reconciles from Git
6. **Heavy stateful workloads** (Postgres, Kafka, etc.): guidance on app-level backup layered on top of the platform mechanism — not "we back up the PVC and pray".
7. **Observability**: backup job status, last successful backup, restore test results visible to platform team without console access (dashboard or CR status).
8. **Tagging / cost attribution**: backup costs traceable per tenant or per tier.
9. **No StorageClass coupling for selection**: backup tier is a property of the workload / namespace, not of the storage layer (workloads must be free to use any performance-tuned SC).
## Success criteria
- A new tenant opts into a backup tier via a single GitOps change.
- A DR drill cross-region completes without manual AWS console intervention.
- Backup coverage and recovery scope are documented and reproducible.
- Failure modes (snapshot fails, region unavailable, restore conflict) have documented playbooks.
## Out of scope for this issue
- Choice of backup engine — to be evaluated as a separate design exercise.
- Application-consistency for specific stateful workloads — handled by app-level operators.
- Backup of EKS control-plane configuration (cluster definition, node groups) — covered by the GitOps + IaC layer.
## Related work
- Draft PR #633 — DR kit RGDs (BackupPolicy + StorageClass).
cc @allamand
Contributor guide
Research direction
Start by reading the functional requirements in this issue and draft PR #633 for the related DR kit RGDs. Evaluate the backup-engine options separately, then map the selected design to tier enrollment, restore workflows, coverage, observability, cost attribution, and failure playbooks. Done means the requirements are implemented and the cross-region drill is reproducible without console steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100