Upload pipeline retry/delete races: DoesNotExist / FK / duplicate-key when pipeline or steps removed mid-flight
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
When a VCF `UploadPipeline` or its `UploadStep`s are deleted or retried while Celery tasks still hold their ids, the tasks blow up. Seen on **variantgrid.com prod** (vgaws) and the SA Path VG3 box.
**Symptoms (grouped Rollbar errors):**
- `UploadStep.objects.get(pk)` → `DoesNotExist` in `load_upload_step` — **80 occurrences** (`upload/tasks/vcf/import_vcf_step_task.py:55`)
- `UploadPipeline.DoesNotExist` in `schedule_pipeline_stage_steps` (`import_vcf_step_task.py:171`)
- `ForeignKeyViolation` inserting `uploadedvcf`/`uploadstep` referencing a deleted `upload_pipeline` (`upload/vcf/vcf_import.py:274`, `upload/vcf/bulk_genotype_vcf_processor.py:457`)
- `UniqueViolation` duplicate key on `cohortgenotype` during `COPY` on re-import (`upload/vcf/sql_copy_files.py:49`)
Root cause: bare `.get()` / inserts assume the pipeline+steps still exist; no handling for concurrent delete/retry. Related: #1670 (closed). (#1534 is a separate partitioning epic.)
**Rollbar:**
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/6946
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/6952
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/7054
- https://app.rollbar.com/a/jimmy.andrews/fix/item/VariantGrid/5806
_(Filed from a 6-month Rollbar triage.)_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.