SACGF / SACGF/variantgrid

Upload pipeline retry/delete races: DoesNotExist / FK / duplicate-key when pipeline or steps removed mid-flight

Open
#1,593 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude

When a VCF `UploadPipeline` or its `UploadStep`s are deleted or retried while Celery tasks still hold their ids, the tasks blow up. Seen on **variantgrid.com prod** (vgaws) and the SA Path VG3 box.

**Symptoms (grouped Rollbar errors):**
- `UploadStep.objects.get(pk)` → `DoesNotExist` in `load_upload_step` — **80 occurrences** (`upload/tasks/vcf/import_vcf_step_task.py:55`)
- `UploadPipeline.DoesNotExist` in `schedule_pipeline_stage_steps` (`import_vcf_step_task.py:171`)
- `ForeignKeyViolation` inserting `uploadedvcf`/`uploadstep` referencing a deleted `upload_pipeline` (`upload/vcf/vcf_import.py:274`, `upload/vcf/bulk_genotype_vcf_processor.py:457`)
- `UniqueViolation` duplicate key on `cohortgenotype` during `COPY` on re-import (`upload/vcf/sql_copy_files.py:49`)

Root cause: bare `.get()` / inserts assume the pipeline+steps still exist; no handling for concurrent delete/retry. Related: #1670 (closed). (#1534 is a separate partitioning epic.)

**Rollbar:**
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/6946
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/6952
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/7054
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/5806

_(Filed from a 6-month Rollbar triage.)_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.