Dispatch to SV annotation pipeline by variant length, not symbolic representation
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
## Problem
Annotation pipeline dispatch keys on whether a variant is *stored symbolically*, not on its length:
- `pipeline_type_variant_q` (`annotation/annotation_version_querysets.py:34`) — `Q(svlen__isnull=False)`
- `AnnotationRun.get_for_variant` (`annotation/models/models.py:1146`) — `variant.is_symbolic`
The intent was that variants above the length threshold go to the SV pipeline. But `svlen` is only set when `as_internal_symbolic` converts a variant, and that gates on **shape** before it ever checks size:
```python
if ref_length == 1: # insertion
if alt_length >= min_symbolic_alt_size: # -> , only if HGVS says 'dup'
elif alt_length == 1 and self.alt == self.ref[0]: # pure deletion
if ref_length >= min_symbolic_alt_size: # ->
elif ref_length == alt_length: # substitution
if ref_length > min_symbolic_alt_size: # -> , only if ref == revcomp(alt)
```
A delins matches none of the three branches, so it never reaches a size comparison at all — it falls through with `svlen=None` and is dispatched to the **standard** pipeline however long it is. Same for insertions HGVS doesn't call a `dup`, and for same-length substitutions that aren't inversions.
The `Sequence` model docstring already records the consequence:
> Even after the introduction of symbolic alts, there are still some really long sequences in here (~10kb) due to large substitutions which we don't represent symbolically
## Evidence
Database-wide:
```
explicit (non-symbolic) sequences >=1000bp : 11,592
variants using them (sample of 5,000 seqs) : 5,043
svlen IS NULL -> STANDARD pipeline : 5,043 (100%)
svlen set -> SV pipeline : 0
```
Not one long explicit variant is routed to the SV pipeline. Examples pulled from standard-pipeline dumps:
| variant | ref -> alt | dump |
|---|---|---|
| `17:39201325` | 1009 -> 12 bp | run 34819 standard |
| `22:46191234` | 1 -> 1331 bp | run 22210 standard (GRCh37) |
| `11:6617775` | 1000 -> 1 bp | run 34818 standard |
## Impact
These variants miss everything the SV lane provides:
- `ANNOTATION_VEP_BUFFER_SIZE` 250 rather than 500
- the #1657 pyBigWig conservation path, rather than four O(span) bigWig `--custom type=overlap` tracks
- AnnotSV
- the `ANNOTATION_VEP_SV_MAX_SIZE` cap
AnnotationRun 34818 (GRCh38, standard, 6,250 variants) has OOM-killed VEP three times — 5.3 GB and 7.3 GB RSS. Both were global OOMs that took the box down with them (2026-08-10 07:08:54 and 2026-08-11 01:18:10), stalling the other concurrent runs and causing knock-on failures in 32515 / 32517 / 32525 / 34819.
Its memory profile tracks large-indel load rather than region sparsity. Per 500-variant buffer:
| buffer | distinct 1Mb regions | n>=400bp | n>=50bp |
|---|---|---|---|
| 1 | 68 | 103 | 430 |
| 2 | 47 | 103 | 410 |
| 3 | 35 | 70 | 334 | <- VEP dies here, all three attempts |
| 4-5 | 48, 45 | 30, 36 | 137, 167 |
| 6-13 | 27-64 | 0 | 1-7 |
Region count is flat across every buffer and buffer 3 is on the low side, so the region cache isn't the driver. The large indels are front-loaded into buffers 1-3 and VEP dies at the end of that cluster. Note this is inference from where the death point lands, not a controlled measurement — worth confirming with a re-run of 34818 with those variants stripped.
Longest variant in that run is 1000 bp, so this is about aggregate large-indel load in a buffer, not any single huge variant.
## Proposed change
Add a length gate to `pipeline_type_variant_q` so a variant above the threshold goes to the SV pipeline regardless of how it happens to be stored:
```python
q_sv = Variant.get_symbolic_q() | Q(= threshold>)
```
Use the current `VARIANT_SYMBOLIC_ALT_SIZE` (1000) as the threshold. Lowering that to 50 is a separate change and should be considered on its own.
## Implementation notes
1. `AnnotationRun.get_for_variant` needs the same predicate. Otherwise a newly created long delins gets dumped into the SV lane while `get_for_variant` looks for its run under `STANDARD` and returns the wrong run or none. The `pipeline_type_variant_q` docstring already flags that these two must agree — today they agree only because both mean "symbolic".
2. The predicate has to be cheap on the variant table. `end` is a stored column, so `end - position` gives ref span without a join; insertions need `Length("alt__seq")` through the `Sequence` FK, which isn't indexed. Likely wants a functional index or a stored length column.
3. Variants already annotated under the standard pipeline will now belong to the SV pipeline. Needs a decision on whether they are re-annotated or left as they are — `AnnotationRangeLock` / `AnnotationRun` accounting is per pipeline type, so the counts move too.
Related: #1656 (introduced `pipeline_type_variant_q`), #1657 (pyBigWig conservation for SVs)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with annotation/annotation_version_querysets.py:34 and AnnotationRun.get_for_variant in annotation/models/models.py:1146, then inspect Variant.get_symbolic_q and the existing docstring. Trace how end, position, and alt__seq are stored and queried before deciding how to make the length predicate cheap. Done means both dispatch paths agree, relevant indexing or storage is addressed, and the treatment of existing standard-pipeline runs is decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100