Flagsmith / Flagsmith/flagsmith

Race conditions in delete_environment and delete_feature tasks cause deadlocks and FK violations

Open
#6,878 2 comments 0 reactions 0 assignees View on GitHub
api
Dominant language
Python
Stars
6.6k
Forks
567
Avg merge
1d 13h
Merged PRs (30d)
121

Description

Multiple Sentry issues report deadlocks, FK constraint violations, and `TransactionManagementError` in the `delete_environment` and `delete_feature` async tasks. These are all caused by concurrent deletion of related objects via the `softdelete` library, where parallel task processor threads compete for locks on the same rows.

## Sentry Issues

- **FLAGSMITH-API-5JR** — `OperationalError: deadlock detected` in `delete_environment` (deadlock on `features_featuresegment`)
- **FLAGSMITH-API-5JJ** — `IntegrityError: FK violation` in `delete_environment` (`feature_id` not present in `features_feature`)
- **FLAGSMITH-API-5JH** — `TransactionManagementError` in `delete_environment` (cascading from deadlock on `feature_versioning_environmentfeatureversion`)
- **FLAGSMITH-API-5JF** — `OperationalError: deadlock detected` in `delete_feature` (deadlock on `features_featuresegment`)
- **FLAGSMITH-API-5JE** — `IntegrityError: FK violation` in `delete_feature` (`identity_id` not present in `environments_identity`)

## Root Cause

The `softdelete` library's `_do_delete` method iterates over related objects and deletes them individually. When `delete_environment` and `delete_feature` tasks run concurrently (e.g. deleting a project triggers both), they compete for locks on shared related objects like `FeatureSegment` and `FeatureState`, causing deadlocks and FK violations at commit time.

All stacktraces originate from:
- `environments/tasks.py:62` — `Environment.objects.get(id=environment_id).delete()`
- `features/tasks.py:162` — `Feature.objects.get(pk=feature_id).delete()`

## Suggested Approach

Consider:
1. Serialising delete operations for related entities (e.g. locking at the project level)
2. Adding retry logic with backoff for deadlock errors
3. Ordering deletions to avoid lock contention between concurrent tasks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.