EbookFoundation / EbookFoundation/regluit-provisioning
Cattle, not pets: scheduled from-scratch rebuild of test + state inventory + logs documented in the playbook
- Dominant language
- Shell
- Stars
- 6
- Forks
- 8
- Avg merge
- 1h 13m
- Merged PRs (30d)
- 2
Description
## Why
2026-09-14: `setup-prod.yml` died at task 1 mid-deploy (apt `python3.8` no longer exists; hardcoded in 17 places; prod's venv is 3.12). Root cause is not the pin — it's that prod was built in June from a provisioning branch that was never merged, and the playbooks are only exercised when something already needs deploying. Playbooks rot when they aren't run.
Raymond, 2026-09-14: *"we easily let our migration playbooks get stale and we need a more robust way to keep them working well. In many ways what we want is a regular build of our servers, treating them more like 'cattle than pets'. Important part though is making sure that we preserve important cumulated state on the servers -- logs, etc."*
## Definition of done
1. **State inventory** — one documented table in this repo: for everything on prod/test, is it *build product* (reproducible from repo + playbook) or *accumulated state* (must be preserved/shipped)? Known state today: RDS (off-box ✅), S3 (off-box ✅), vault secrets (✅), **logs** (on-box ❌ — see #60), TLS certs (#67), anything hand-edited since June.
2. **Logs** — which logs we keep, for how long, and where they go off-box (CloudWatch Logs or S3), **documented in the playbook/README in terms Eric understands** (Eric's only requirement: the right logs are saved and he knows where they are).
3. **Test is rebuilt from scratch on a schedule** (monthly) by the playbook, as the standing proof that it works — this replaces any "dry-run rot check". A rebuild that fails is a finding, not an outage.
4. **Prod follows the Blue/Green shape** already used for the DB: build fresh from playbook → smoke → switch → keep old 24 h → delete. #56 is redefined to this.
5. Guardrail: same tooling (Ansible + Ubuntu). Not a platform rewrite.
## Related
#56 (recreate prod & test deliberately), #60 (preserve pre-6/18 download logs), #67 (certbot regresses on rebuild), #66 (group_vars/test incomplete), plan PR #73.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with setup-prod.yml, the playbook/README, and group_vars/test to map the current provisioning flow and test environment. Inventory build products versus accumulated state, including logs and TLS certificates, then document retention and off-box destinations. Done means a scheduled monthly test rebuild and a documented Blue/Green production flow matching the stated guardrails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, shell, ubuntu
- Domain
- cloud, devops, documentation, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100