EbookFoundation / EbookFoundation/regluit-provisioning

Preserve pre-6/18 download logs from old prod (prod7_py3) — snapshot taken, storage destination TBD

Open
#60 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
6
Forks
8
Avg merge
1h 13m
Merged PRs (30d)
2

Description

## Summary

The pre-cutover unglue.it production download logs (`/var/log/regluit/downloads.log` from **before the 2026-06-18 Django 4.2 cutover**) **were preserved** and are now backed up in an EBS snapshot. This issue records what exists, where, and the one open decision: **where to permanently store a copy** of the recovered logs.

Prompted by Eric asking whether the pre-6/18 download logs in `/var/log/regluit` survived the cutover.

## Background

The Django 4.2 production cutover was a blue/green swap:

- **`prod-green`** (`i-0b0d4da7db10adee2`) launched **2026-06-18 00:02 UTC**, took over the live Elastic IP, and is the current production box.
- **`prod7_py3`** (`i-038105b781b514601`) — the **old (pre-cutover) production box** — was **stopped** (not terminated) at **2026-06-18 05:16 UTC**, ~5 h after green came up.

Because the old box was parked rather than terminated, its root disk (with `/var/log/regluit`) was still intact.

## Investigation (read-only)

- `prod7_py3` is **stopped, not terminated**; its 16 GB root volume `vol-0dadb3ff583119309` was still attached and healthy.
- **No prior backup existed** — no EBS snapshot of that volume, and no log shipping to S3 (`unglueit-logs` holds only a 2024 DB export; `unglueit/logs/` is 2016-era S3 access logs; `gluejar-logs` is empty). The parked disk was the **sole copy**.
- The volume had **`DeleteOnTermination = true`** — i.e. terminating `prod7_py3` would have destroyed the only copy.

To inspect without booting the old box (so none of its Celery jobs / email senders could fire), I:
1. Took a snapshot of the root volume, then
2. Created a volume from the snapshot, attached it **read-only** to the running test box, mounted it, inventoried `/var/log/regluit`, and
3. Detached + deleted the temp volume. **`prod7_py3` was never powered on.** Test box returned to its original state.

## What's on the old disk (`prod7_py3:/var/log/regluit`)

| File | Size | Coverage (from actual contents) |
|---|---|---|
| **`downloads.log`** | 18.9 MB | **2026-05-11 09:11:30 → 2026-06-18 00:19:09** — **577,337 download events** |
| `unglue.it.log` | 3.6 MB | app log, last write 2026-06-18 02:50 |
| `doab-harvest.log` | 72 KB | last write 2026-06-18 04:30 |

Notes:
- `downloads.log` format is `timestamp : work_id` — **no secrets**.
- It is a **single un-rotated file starting 2026-05-11**. There are **no older download archives** on the disk (no `downloads.log.1/.2`, no gzipped download logs). Anything **before 2026-05-11** had already rotated away *before* the cutover and is **not recoverable from this disk** — this is not something the cutover lost. (There's no logrotate config for regluit; Django's `RotatingFileHandler` rotates at ~20 MB and this file was still 18.9 MB, just under threshold, so it never rotated and no backups were kept.)

## Continuity with the current box

- `prod7_py3` `downloads.log` **ends** `2026-06-18 00:19:09`.
- `prod-green` `downloads.log` **begins** `2026-06-18 00:19:19`.
- So the two boxes together give **continuous download coverage from 2026-05-11 onward**, with only a **~10-second seam** at the cutover.

## Backup created

- **Snapshot `snap-0b9d1d7dec2c6b95f`** (16 GB, us-east-1, Gluejar account) — full copy of `prod7_py3`'s root disk. Tagged `Name=prod7-preserve-download-logs-20260702`, `purpose=preserve-pre-cutover-logs`.
- This **permanently preserves** the logs and **closes the `DeleteOnTermination` risk** — the logs now survive even if `prod7_py3` is eventually terminated.

## Open decision — where to store the recovered copy

No log files have been copied anywhere yet. Options:

- **S3** — e.g. `s3://unglueit-logs/prod7-pre-cutover/` or a dedicated bucket (durable, easy to hand off). *Suggested default.*
- **Onto `prod-green`** — alongside the current logs so the full history lives in one place.
- **Local / hand to Eric directly** — it's only ~19 MB.

Once a destination is chosen, the copy can be pulled straight from snapshot `snap-0b9d1d7dec2c6b95f` (no need to boot `prod7_py3`).

## Cleanup / current state

- ✅ Snapshot `snap-0b9d1d7dec2c6b95f` retained (the backup).
- ✅ Temp inspection volume detached + deleted.
- ✅ Test box back to original state; `prod7_py3` still **stopped**, never booted.

Related: download-log audit / reconstruction in Gluejar/regluit#1140.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with snapshot snap-0b9d1d7dec2c6b95f and the recovered /var/log/regluit/downloads.log on prod7_py3; review the listed S3, prod-green, and local-storage options and confirm the destination with the project owner. Done means copying the recovered logs to the chosen destination without booting prod7_py3 and recording where the copy lives.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.