EbookFoundation / EbookFoundation/regluit-provisioning

Cattle, not pets: scheduled from-scratch rebuild of test + state inventory + logs documented in the playbook

Open
#74 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
6
Forks
8
Avg merge
1h 13m
Merged PRs (30d)
2

Description

## Why
2026-09-14: `setup-prod.yml` died at task 1 mid-deploy (apt `python3.8` no longer exists; hardcoded in 17 places; prod's venv is 3.12). Root cause is not the pin — it's that prod was built in June from a provisioning branch that was never merged, and the playbooks are only exercised when something already needs deploying. Playbooks rot when they aren't run.

Raymond, 2026-09-14: *"we easily let our migration playbooks get stale and we need a more robust way to keep them working well. In many ways what we want is a regular build of our servers, treating them more like 'cattle than pets'. Important part though is making sure that we preserve important cumulated state on the servers -- logs, etc."*

## Definition of done
1. **State inventory** — one documented table in this repo: for everything on prod/test, is it *build product* (reproducible from repo + playbook) or *accumulated state* (must be preserved/shipped)? Known state today: RDS (off-box ✅), S3 (off-box ✅), vault secrets (✅), **logs** (on-box ❌ — see #60), TLS certs (#67), anything hand-edited since June.
2. **Logs** — which logs we keep, for how long, and where they go off-box (CloudWatch Logs or S3), **documented in the playbook/README in terms Eric understands** (Eric's only requirement: the right logs are saved and he knows where they are).
3. **Test is rebuilt from scratch on a schedule** (monthly) by the playbook, as the standing proof that it works — this replaces any "dry-run rot check". A rebuild that fails is a finding, not an outage.
4. **Prod follows the Blue/Green shape** already used for the DB: build fresh from playbook → smoke → switch → keep old 24 h → delete. #56 is redefined to this.
5. Guardrail: same tooling (Ansible + Ubuntu). Not a platform rewrite.

## Related
#56 (recreate prod & test deliberately), #60 (preserve pre-6/18 download logs), #67 (certbot regresses on rebuild), #66 (group_vars/test incomplete), plan PR #73.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with setup-prod.yml, the playbook/README, and group_vars/test to map the current provisioning flow and test environment. Inventory build products versus accumulated state, including logs and TLS certificates, then document retention and off-box destinations. Done means a scheduled monthly test rebuild and a documented Blue/Green production flow matching the stated guardrails.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, shell, ubuntu
Domain
cloud, devops, documentation, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.