TLS: make certbot the single cert mechanism, managed by provisioning (rebuild currently regresses it)

## Problem

There are **two competing certificate mechanisms** for the web hosts, and the one provisioning believes in is not the one actually serving traffic.

**(A) Certbot on the box** — systemd timer, twice daily. This issues and serves the live certificate today, from `/etc/letsencrypt/live/unglue.it/`.

**(B) This repo** — `roles/regluit_prod/tasks/certs.yml` uses the `acme_certificate` module with HTTP-01 challenges and vault-encrypted key material, copying certs to `/etc/ssl`. It renews only when someone runs the playbook.

**Consequence: a rebuild or full `setup-prod.yml` run could regress a working TLS setup**, and a fresh box cannot be built faithfully at all, because it lacks the `/etc/letsencrypt` lineage Apache is configured to read.

## Why now

unglue.it's automatic renewal was **broken from 2026-06-17 until 2026-08-18** and nobody noticed for two months. The cert was pre-issued with `--dns-route53`, so certbot stored an authenticator needing Route 53 credentials the box doesn't have; every renewal failed with `Unable to locate credentials`. Expiry (2026-09-15) would have landed inside the relaunch window.

Fixed on production 2026-08-18 by switching to HTTP-01 webroot and adding an Apache-reload deploy hook — but **that fix currently lives only as box-local state**, which is exactly the problem this issue exists to close. Background: Gluejar/regluit#1075, `Gluejar/INCIDENT_2026-08-18_tls_renewal_fix.md`.

## Recommendation

**Certbot should win — but managed by Ansible, not left as undocumented box-local drift.**

Certbot owns issuance and automatic renewal; provisioning installs and configures it. This gives one ordinary mechanism a non-expert can reason about (`certbot.timer`), renewal independent of deploys, **no Route 53 credentials on the web server**, and the smallest blast radius. Remove the controller-side `acme_certificate` lifecycle.

## Concrete changes

1. Replace `manage_certs` with an explicit `tls_manager: certbot_webroot`; define `tls_domains` per environment.
2. **Retire the controller-side ACME/key-copy workflow** in `certs.yml` — the decrypted account key, the CSR, the `/etc/ssl` copies, and `force: yes`.
3. Install certbot; create a **dedicated root-owned webroot** (e.g. `/var/lib/letsencrypt/.well-known/acme-challenge`) and render a **direct Apache mapping on port 80**.
4. **Split Apache bootstrap ordering**: bring up an HTTP-only challenge vhost, obtain the initial cert, then render and enable the TLS vhost. The present Apache-before-cert ordering cannot bootstrap a fresh box.
5. Provision the root-owned deploy hook (`configtest` then `systemctl reload apache2`), and enable/start `certbot.timer`.
6. Document the **blue/green wrinkle**: before DNS/EIP moves to a replacement host, HTTP-01 validation still reaches the *old* host. A cutover needs either secure transfer of the still-valid certbot lineage, or a short HTTP bootstrap immediately after the address moves — without leaving standing Route 53 credentials anywhere.

## Bugs found in `certs.yml` while reviewing

- **`force: yes` on the first ACME call** overrides the stated `remaining_days: 45`, so a full run may reissue redundantly every time (`certs.yml:48`).
- **The "delete decrypted files" task uses `state: file`, not `state: absent`** (`certs.yml:170`) — so the **decrypted ACME account key and CSR are left on disk**. That is a security defect independent of everything else here.
- `main.yml:159` imports `certs.yml` unconditionally.
- The role writes a second, unused certificate/key pair under `/etc/ssl`.

Partially overlaps #30 (certs.yml cleanup), which should probably be folded into or closed by this work.

## The fragility this must also fix

Production's current webroot is `/var/www/static`, reached via `/.well-known/acme-challenge/` → 301 → HTTPS → Django → 302 → `/static/…`. Let's Encrypt follows redirects, so it works — but renewal now transitively depends on **Apache, WSGI, Django, the URL route, and maintenance mode being off**. The maintenance-mode rewrite returns **503 for `/.well-known/…`** before Django can redirect, so *a renewal firing during maintenance would fail*.

Note that simply adding an `Alias` is **not** sufficient: Apache's `Redirect` takes precedence over `Alias`. The catch-all `Redirect permanent /` must be replaced with a rule that **excludes** the ACME path.

## Acceptance criteria

- [ ] Direct HTTP-01 challenge path returns 200 on port 80 **without involving Django**, and works with the site in maintenance mode
- [ ] Real certificate served on apex and `www`; serial matches the certbot lineage
- [ ] `certbot renew --dry-run --run-deploy-hooks` succeeds against the **stored** config
- [ ] `certbot.timer` enabled and scheduled
- [ ] Deploy hook is root-owned, guarded by `configtest`, and propagates non-zero on failure
- [ ] A **from-scratch** provision of test.unglue.it obtains and serves a cert with no manual steps
- [ ] Decrypted key material is genuinely removed (`state: absent`)
- [ ] Applied to test **and** production, converging today's manual production state into provisioning

## Monitoring (separate but related)

Daily **external** TLS-expiry check for `unglue.it`, `www.unglue.it`, `test.unglue.it` — alert under 30 days, repeat until acknowledged. External catches failure-to-renew, failure-to-reload, wrong SNI, and stale-cert-still-served. Secondary: alert on `certbot.service` unit failure.

Findings reviewed by Codex (gpt-5.4).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01SGNAvPnAPCuS2vFAg6AiBB

Start with roles/regluit_prod/tasks/certs.yml and main.yml:159, then trace Apache ordering in setup-prod.yml and the current certbot configuration on the hosts. Validate the challenge path and provisioning flow on test.unglue.it, including certbot renew --dry-run --run-deploy-hooks. Done means a from-scratch test provision and production convergence satisfy every TLS acceptance criterion without manual steps.

Written by the indexing model from the issue text.