EbookFoundation / EbookFoundation/regluit-provisioning
Preserve pre-6/18 download logs from old prod (prod7_py3) — snapshot taken, storage destination TBD
- Dominant language
- Shell
- Stars
- 6
- Forks
- 8
- Avg merge
- 1h 13m
- Merged PRs (30d)
- 2
Description
## Summary
The pre-cutover unglue.it production download logs (`/var/log/regluit/downloads.log` from **before the 2026-06-18 Django 4.2 cutover**) **were preserved** and are now backed up in an EBS snapshot. This issue records what exists, where, and the one open decision: **where to permanently store a copy** of the recovered logs.
Prompted by Eric asking whether the pre-6/18 download logs in `/var/log/regluit` survived the cutover.
## Background
The Django 4.2 production cutover was a blue/green swap:
- **`prod-green`** (`i-0b0d4da7db10adee2`) launched **2026-06-18 00:02 UTC**, took over the live Elastic IP, and is the current production box.
- **`prod7_py3`** (`i-038105b781b514601`) — the **old (pre-cutover) production box** — was **stopped** (not terminated) at **2026-06-18 05:16 UTC**, ~5 h after green came up.
Because the old box was parked rather than terminated, its root disk (with `/var/log/regluit`) was still intact.
## Investigation (read-only)
- `prod7_py3` is **stopped, not terminated**; its 16 GB root volume `vol-0dadb3ff583119309` was still attached and healthy.
- **No prior backup existed** — no EBS snapshot of that volume, and no log shipping to S3 (`unglueit-logs` holds only a 2024 DB export; `unglueit/logs/` is 2016-era S3 access logs; `gluejar-logs` is empty). The parked disk was the **sole copy**.
- The volume had **`DeleteOnTermination = true`** — i.e. terminating `prod7_py3` would have destroyed the only copy.
To inspect without booting the old box (so none of its Celery jobs / email senders could fire), I:
1. Took a snapshot of the root volume, then
2. Created a volume from the snapshot, attached it **read-only** to the running test box, mounted it, inventoried `/var/log/regluit`, and
3. Detached + deleted the temp volume. **`prod7_py3` was never powered on.** Test box returned to its original state.
## What's on the old disk (`prod7_py3:/var/log/regluit`)
| File | Size | Coverage (from actual contents) |
|---|---|---|
| **`downloads.log`** | 18.9 MB | **2026-05-11 09:11:30 → 2026-06-18 00:19:09** — **577,337 download events** |
| `unglue.it.log` | 3.6 MB | app log, last write 2026-06-18 02:50 |
| `doab-harvest.log` | 72 KB | last write 2026-06-18 04:30 |
Notes:
- `downloads.log` format is `timestamp : work_id` — **no secrets**.
- It is a **single un-rotated file starting 2026-05-11**. There are **no older download archives** on the disk (no `downloads.log.1/.2`, no gzipped download logs). Anything **before 2026-05-11** had already rotated away *before* the cutover and is **not recoverable from this disk** — this is not something the cutover lost. (There's no logrotate config for regluit; Django's `RotatingFileHandler` rotates at ~20 MB and this file was still 18.9 MB, just under threshold, so it never rotated and no backups were kept.)
## Continuity with the current box
- `prod7_py3` `downloads.log` **ends** `2026-06-18 00:19:09`.
- `prod-green` `downloads.log` **begins** `2026-06-18 00:19:19`.
- So the two boxes together give **continuous download coverage from 2026-05-11 onward**, with only a **~10-second seam** at the cutover.
## Backup created
- **Snapshot `snap-0b9d1d7dec2c6b95f`** (16 GB, us-east-1, Gluejar account) — full copy of `prod7_py3`'s root disk. Tagged `Name=prod7-preserve-download-logs-20260702`, `purpose=preserve-pre-cutover-logs`.
- This **permanently preserves** the logs and **closes the `DeleteOnTermination` risk** — the logs now survive even if `prod7_py3` is eventually terminated.
## Open decision — where to store the recovered copy
No log files have been copied anywhere yet. Options:
- **S3** — e.g. `s3://unglueit-logs/prod7-pre-cutover/` or a dedicated bucket (durable, easy to hand off). *Suggested default.*
- **Onto `prod-green`** — alongside the current logs so the full history lives in one place.
- **Local / hand to Eric directly** — it's only ~19 MB.
Once a destination is chosen, the copy can be pulled straight from snapshot `snap-0b9d1d7dec2c6b95f` (no need to boot `prod7_py3`).
## Cleanup / current state
- ✅ Snapshot `snap-0b9d1d7dec2c6b95f` retained (the backup).
- ✅ Temp inspection volume detached + deleted.
- ✅ Test box back to original state; `prod7_py3` still **stopped**, never booted.
Related: download-log audit / reconstruction in Gluejar/regluit#1140.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with snapshot snap-0b9d1d7dec2c6b95f and the recovered /var/log/regluit/downloads.log on prod7_py3; review the listed S3, prod-green, and local-storage options and confirm the destination with the project owner. Done means copying the recovered logs to the chosen destination without booting prod7_py3 and recording where the copy lives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100