SACGF / SACGF/variantgrid

Analysis benchmarking: capture/store/auto-run reload snapshots

Open
#1,552 0 comments 0 reactions 0 assignees View on GitHub
Analysis optimisation
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 22m
Merged PRs (30d)
40

Description

🤖 Written by Claude

Spec: [`claude/analysis_benchmarking_plan.md`](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md) — every locked decision, schema, and open item is in there.

## Motivation in one paragraph

The `random_page_cost = 1.1` win that closed #1546 was discovered by running `profile_analysis_nodes --planner-diagnostic` manually on prod and eyeballing a CSV. A regression in the *other* direction — a query plan that accidentally falls back to a seq-scan after a deploy, or a `CohortGenotypeStats` cache that quietly stops getting hit — would today only surface as a user complaint, with no easy way to check "is this analysis slower than it was last week?" This feature ships an admin-only "Benchmark" button on the analysis page that captures a reload-time snapshot (wall + CPU + per-node breakdown + git/env provenance), plus a Celery Beat task that runs an opt-in bucket of analyses every Saturday 03:00. Diff any two snapshots in the UI, catch drift before users do.

## What's already landed (preconditions)

- `NodeVersion` and `NodeCount` are `TimeStampedModel` (analysis migration `0101`). NodeCount.created cascades per NodeVersion, so it automatically tracks the latest load batch — used by the retrospective `get_cpu_and_walltime` inspector and as a sanity check inside the snapshot writer.
- `profile_analysis_nodes --planner-diagnostic` exists for ad-hoc settings sweeps; the snapshot feature reuses its `_PgSessionSettings` context manager pattern for the optional sweep-mode stretch goal.

## What this issue covers

Per [the plan](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md):

- Three new models (`AnalysisBenchmarkSnapshot`, `AnalysisBenchmarkNodeRow`, `AnalysisAutoBenchmarkOptIn`) — see plan §Schema.
- Synchronous reload-and-wait wrapper around the existing Celery-driven node update pipeline — see plan §"Reload mechanics" and open item #1.
- Admin-only UI: button on analysis page, snapshot history view, snapshot detail view, comparison view — see plan §"Admin UI".
- Celery Beat schedule firing Saturday 03:00 + dedicated `benchmark_workers` queue — see plan §"Celery Beat auto bucket". Saturday 03:00 was chosen because it's empty in the existing schedule survey (DB activity is at 06:00/19:00, weekly classification email is Mon 10:00, RDS snapshot windows are weekday early hours).
- Management command for ad-hoc / CI sweeps — see plan §"Management command".

## Out of scope (clean follow-up issues if/when they're needed)

The plan §"Stretch features — not v1" lists these explicitly with rationale:

- Alerting on `wall_seconds > 1.5 × trailing_30_day_average`
- Auto-EXPLAIN capture on regressed nodes
- Settings sweep mode (mirrors `--planner-diagnostic` per-analysis)
- Multiple Beat schedules per opt-in bucket
- CI integration (sentinel analyses run on test DB, fail build on regression)

## Open items the implementer needs to confirm before coding

[Plan §"Open items / things to confirm before implementation"](https://github.com/SACGF/variantgrid/blob/master/claude/analysis_benchmarking_plan.md#open-items--things-to-confirm-before-implementation) lists five — synchronous reload wrapper shape, PG settings whitelist completeness, benchmark queue concurrency, NULL `run_by` template handling, CGC version source-of-truth.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with claude/analysis_benchmarking_plan.md, especially the Schema, Reload mechanics, Admin UI, Celery Beat auto bucket, and open-items sections. Then inspect profile_analysis_nodes --planner-diagnostic and the existing Celery-driven node update pipeline. Done means the three models, admin views, scheduled opt-in runs, dedicated queue, and management command work together as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, database, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.