VEPVersionMismatchError: fail once, then stop scheduling runs against that annotation version

🤖 Written by Claude

## Problem

A VEP version mismatch is a property of the deployment (the installed VEP no longer matches the `VariantAnnotationVersion`), so once it is true it is true for *every* run against that VAV. But we only discover it per-run, inside `annotate_variants`, so the scheduler keeps creating runs, the dispatcher keeps launching them, and each one fails identically and reports to Rollbar.

Current Rollbar (Jul–Aug 2026):

| Item | Environment | Events | Last |
|---|---|---|---|
| 7460 `VEPVersionMismatchError … KEY: 'vep' annotation: '103', vcf: '116'` | vg4test | 231 | 2026-08-05, still firing |
| 7454 `Annotation pipeline run 78407` (the same failures' wrapper) | vg4test | 240 | 2026-08-05 |
| 7346 `VEPVersionMismatchError … KEY: 'gencode_subset' annotation: 'None', vcf: 'primary'` | vgaws | 193 | 2026-07-07 |
| 3045 `Annotation pipeline run 17168` (wrapper) | vgaws | 288 | 2026-07-30 |

That is ~950 events describing two configuration problems. It buries everything else in Rollbar, and it burns annotation worker slots re-dumping variants for runs that cannot succeed.

(#1642 covered item 7346 and was closed 2026-07-03, but the item kept firing to 2026-07-07 — worth confirming the vgaws VAV was actually rebuilt.)

## Where it happens now

* `vep_check_command_line_version_match()` (`annotation/vep_annotation.py:567`) is called from `dump_and_annotate_variants` (`annotation/tasks/annotate_variants.py:459`), i.e. once per run, immediately before dumping.
* The raise is `_vep_check_version_match` (`annotation/vep_annotation.py:555`).
* The exception unwinds into the generic handler at `annotation/tasks/annotate_variants.py:280`, which reports `Annotation pipeline run ` at error level — hence each mismatch produces two Rollbar events.

## Proposal — latch the failure on the VAV, gate the dispatcher

`VariantAnnotationVersion.get_annotation_run_blocker()` (`annotation/models/models.py:794`) already exists for exactly this idea — "runs against this VAV will fail, here's why" — and is already surfaced on the annotation runs page (`annotation/templates/annotation/variant_annotation_runs.html:146`). What it lacks is (a) knowledge of VEP mismatches and (b) any effect on scheduling: it is only consulted *inside* a run, at `annotate_variants.py:245`, after the run has already been created, leased and launched.

1. **Latch.** Add a field to `VariantAnnotationVersion` (e.g. `annotation_blocked_reason` + `annotation_blocked_date`, plus migration). When `annotate_variants` catches `VEPVersionMismatchError`, it records the reason on the VAV and reports to Rollbar **only on the transition into blocked**. First failure stays loud; subsequent ones are silent.

2. **Blocker.** `get_annotation_run_blocker()` returns the latched reason alongside the existing missing-`GeneAnnotationRelease` reason.

3. **Gate.** `_dispatch_for_vav` (`annotation/tasks/annotation_scheduler_task.py:349`) and `_dispatch_sweep` (`:284`) consult the blocker and skip a blocked VAV entirely, so nothing is leased or launched. `annotation_scheduler` likewise stops taking new range locks for it. This ends the fail-after-fail loop at the source rather than making it quieter.

This also fixes the missing-`GeneAnnotationRelease` case, which has the identical shape today.

4. **Unblock explicitly.** A "re-check VEP version" action on the annotation runs page, dispatched to `annotation_workers`, which runs `vep_check_command_line_version_match()` and clears the latch only when it now matches. Keeping the clear conditional on a real check means an operator cannot unblock straight back into the same loop.

## Notes / constraints

* The check has to stay on `annotation_workers`. `get_vep_version()` (`annotation/vep_annotation.py:277`) runs a full VEP invocation on a fake VCF to read the `##VEP=` header — expensive, `@lru_cache`d per process, and only valid on a box with VEP installed. The dispatcher runs on `scheduling_single_worker`, which is not guaranteed to be that box (see #1665). Hence latching a worker-observed result rather than checking from the scheduler.
* Because the version is memoised per process, workers need restarting after a VEP upgrade — the re-check action should account for that (documented, or the task re-derives rather than reusing the cached value).
* Runs already leased when the latch is set will still fail; the loop stops after that wave.
* `gencode_subset` and `distance` are already excluded from comparison as settings-snapshot fields (`annotation/vep_annotation.py:540-546`), so item 7346's shape should no longer be reachable — worth confirming while implementing.

## Related

* #1642 — the vgaws `gencode_subset` instance of this, closed but recurred.
* #1658 / #1660 — annotation run lease / reclaim work, same task file.

Start by tracing the existing blocker flow through annotation/models/models.py, annotation/tasks/annotate_variants.py, and annotation/tasks/annotation_scheduler_task.py, then inspect the annotation runs page and the annotation_workers entry point. Confirm how VEPVersionMismatchError is raised and reported before changing the behavior. Done means mismatches latch on the VAV, blocked VAVs are not dispatched, and an explicit worker-side re-check clears the latch only after a successful match.

Written by the indexing model from the issue text.