Analysis benchmarking: capture/store/auto-run reload snapshots
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 22m
- Merged PRs (30d)
- 40
Description
🤖 Written by Claude
Spec: [`claude/analysis_benchmarking_plan.md`](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md) — every locked decision, schema, and open item is in there.
## Motivation in one paragraph
The `random_page_cost = 1.1` win that closed #1546 was discovered by running `profile_analysis_nodes --planner-diagnostic` manually on prod and eyeballing a CSV. A regression in the *other* direction — a query plan that accidentally falls back to a seq-scan after a deploy, or a `CohortGenotypeStats` cache that quietly stops getting hit — would today only surface as a user complaint, with no easy way to check "is this analysis slower than it was last week?" This feature ships an admin-only "Benchmark" button on the analysis page that captures a reload-time snapshot (wall + CPU + per-node breakdown + git/env provenance), plus a Celery Beat task that runs an opt-in bucket of analyses every Saturday 03:00. Diff any two snapshots in the UI, catch drift before users do.
## What's already landed (preconditions)
- `NodeVersion` and `NodeCount` are `TimeStampedModel` (analysis migration `0101`). NodeCount.created cascades per NodeVersion, so it automatically tracks the latest load batch — used by the retrospective `get_cpu_and_walltime` inspector and as a sanity check inside the snapshot writer.
- `profile_analysis_nodes --planner-diagnostic` exists for ad-hoc settings sweeps; the snapshot feature reuses its `_PgSessionSettings` context manager pattern for the optional sweep-mode stretch goal.
## What this issue covers
Per [the plan](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md):
- Three new models (`AnalysisBenchmarkSnapshot`, `AnalysisBenchmarkNodeRow`, `AnalysisAutoBenchmarkOptIn`) — see plan §Schema.
- Synchronous reload-and-wait wrapper around the existing Celery-driven node update pipeline — see plan §"Reload mechanics" and open item #1.
- Admin-only UI: button on analysis page, snapshot history view, snapshot detail view, comparison view — see plan §"Admin UI".
- Celery Beat schedule firing Saturday 03:00 + dedicated `benchmark_workers` queue — see plan §"Celery Beat auto bucket". Saturday 03:00 was chosen because it's empty in the existing schedule survey (DB activity is at 06:00/19:00, weekly classification email is Mon 10:00, RDS snapshot windows are weekday early hours).
- Management command for ad-hoc / CI sweeps — see plan §"Management command".
## Out of scope (clean follow-up issues if/when they're needed)
The plan §"Stretch features — not v1" lists these explicitly with rationale:
- Alerting on `wall_seconds > 1.5 × trailing_30_day_average`
- Auto-EXPLAIN capture on regressed nodes
- Settings sweep mode (mirrors `--planner-diagnostic` per-analysis)
- Multiple Beat schedules per opt-in bucket
- CI integration (sentinel analyses run on test DB, fail build on regression)
## Open items the implementer needs to confirm before coding
[Plan §"Open items / things to confirm before implementation"](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md#open-items--things-to-confirm-before-implementation) lists five — synchronous reload wrapper shape, PG settings whitelist completeness, benchmark queue concurrency, NULL `run_by` template handling, CGC version source-of-truth.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with claude/analysis_benchmarking_plan.md, especially the Schema, Reload mechanics, Admin UI, Celery Beat auto bucket, and open-items sections. Then inspect profile_analysis_nodes --planner-diagnostic and the existing Celery-driven node update pipeline. Done means the three models, admin views, scheduled opt-in runs, dedicated queue, and management command work together as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, database, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100