aws-samples / aws-samples/appmod-blueprints

[DR] Backup & restore capability for stateful workloads on spoke clusters

Open
#643 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
105
Forks
62
Avg merge
11h 17m
Merged PRs (30d)
76

Description

## Context

The platform currently provisions stateful workloads on spoke EKS clusters but offers no built-in backup/restore mechanism. Operators must bring their own tooling, with no GitOps-driven contract for protection policies. This issue captures the **functional requirements** for adding a backup & restore capability to `appmod-blueprints` spoke clusters. The technical solution (AWS Backup for EKS, Velero, Kasten, or other) is **not yet defined** and will be evaluated separately.

## Functional requirements

1. **Tiered protection policies** declared as code (e.g. gold / silver / bronze) with distinct RPO, retention, and cross-region semantics. Tenants opt into a tier — they don't compose individual backup parameters.
2. **GitOps contract**: tenants enroll a workload via a single declarative manifest (CR / annotation / label — TBD by design). No out-of-band AWS console steps, no platform-team ticket per tenant.
3. **Cross-region disaster recovery**: at least one tier MUST replicate backups to a secondary AWS region.
4. **Restore workflows** documented and testable for:
- in-place restore (same cluster, same namespace)
- cross-region restore (DR cluster in secondary region)
- clone-to-new-environment (stage-to-prod, blue/green, ephemeral envs)
5. **Coverage scope explicit** — what is protected vs what stays out:
- PVC data (crash-consistent at minimum, app-consistent where supported)
- Kubernetes manifests not derivable from Git (e.g. dynamic CRs)
- RBAC, ConfigMaps, Secrets — explicit decision per category
- What is **explicitly out**: secrets re-hydrated by External Secrets Operator, anything Argo CD reconciles from Git
6. **Heavy stateful workloads** (Postgres, Kafka, etc.): guidance on app-level backup layered on top of the platform mechanism — not "we back up the PVC and pray".
7. **Observability**: backup job status, last successful backup, restore test results visible to platform team without console access (dashboard or CR status).
8. **Tagging / cost attribution**: backup costs traceable per tenant or per tier.
9. **No StorageClass coupling for selection**: backup tier is a property of the workload / namespace, not of the storage layer (workloads must be free to use any performance-tuned SC).

## Success criteria

- A new tenant opts into a backup tier via a single GitOps change.
- A DR drill cross-region completes without manual AWS console intervention.
- Backup coverage and recovery scope are documented and reproducible.
- Failure modes (snapshot fails, region unavailable, restore conflict) have documented playbooks.

## Out of scope for this issue

- Choice of backup engine — to be evaluated as a separate design exercise.
- Application-consistency for specific stateful workloads — handled by app-level operators.
- Backup of EKS control-plane configuration (cluster definition, node groups) — covered by the GitOps + IaC layer.

## Related work

- Draft PR #633 — DR kit RGDs (BackupPolicy + StorageClass).

cc @allamand

Contributor guide

Open the contributing guide

Research direction

Start by reading the functional requirements in this issue and draft PR #633 for the related DR kit RGDs. Evaluate the backup-engine options separately, then map the selected design to tier enrollment, restore workflows, coverage, observability, cost attribution, and failure playbooks. Done means the requirements are implemented and the cross-region drill is reproducible without console steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.