Flagsmith / Flagsmith/flagsmith
Deleting an identity while an override is being written orphans the override in environments_v2
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 567
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 124
Description
On an Edge (DynamoDB) project, an identity override can be left behind in `environments_v2` after the identity it belongs to is deleted. The feature's identity-overrides tab then lists an override that does not exist on the identity, local-evaluation SDKs keep applying it while remote evaluation does not, and the row cannot be removed through the API or the dashboard because it is keyed on an identity UUID that no longer resolves.
## Reproduction
Any Edge-enabled environment, two features, and an admin API key. The trigger is adding an identity override while that identity is being deleted — which is what a test suite or automation doing per-run identities hits in the wild.
Using the [CLI](https://docs.flagsmith.com/integrating-with-flagsmith/CLI):
```bash
ENV=
F1=
F2=
IDENT="repro-$(uuidgen)"
# 1. Create an identity carrying an override on F1
# (the by-identifier endpoint creates the identity if it doesn't exist)
flagsmith api "api/v1/environments/environments/$ENV/edge-identities-featurestates" \
-X PUT -f identifier="$IDENT" -F feature=$F1 -F enabled=true
# -> take identity_uuid from the response
UUID=
# 2. Add an override on F2 *while* deleting the identity
flagsmith api "api/v1/environments/environments/$ENV/edge-identities-featurestates" \
-X PUT -f identifier="$IDENT" -F feature=$F2 -F enabled=true &
flagsmith api "api/v1/environments/$ENV/edge-identities/$UUID/" -X DELETE &
wait
# 3. Ask the feature which identities override it
flagsmith api "api/v1/environments/$ENV/edge-identity-overrides?feature=$F2"
```
**Expected:** no override document for `$IDENT`.
**Actual:** `$IDENT` is listed against F2. Deleting the identity again does not clear it, and neither does deleting the override — the row's `identity_uuid` does not resolve to any identity.
Over 20 iterations this orphaned an override on **16 of them**. Running the same three steps sequentially, with no overlap between step 2's two calls, orphans nothing — the overlap is the whole trigger.
Several of the 16 also came back with the identity document present under a *different* `identity_uuid` and an empty `identity_features`, which is the state that makes the dashboard contradict itself: the feature tab lists the override, the identity page says the identity is using environment defaults.
## Cause
Identity writes are a whole-document read-modify-write on `flagsmith_identities` with no concurrency control (`EdgeIdentity.save()` → unconditional `put_item`), and the `environments_v2` changeset is derived by diffing the mutated model against the document *as it was read* (`_get_changes` vs `_initial_state`). A write that lands between another request's read and its write is invisible to that request's diff, so the delete emits `-` only for the overrides it happened to see, and the other override's `environments_v2` document is never removed. Nothing reconciles the two stores afterwards.
## Suggested fix
1. Optimistic concurrency control on the identity document — a version attribute plus a condition expression on `put_item`, retrying on conflict. `DynamoIdentityWrapper.set_system_trait` already uses conditional writes for the same reason. Consistent reads alone would not have prevented any of the 16 reproductions.
2. `ConsistentRead=True` on the `composite_key` read, and resolve uuid → `composite_key` via the GSI before reading the table, rather than deriving `identity_features` from a GSI projection (`get_item_from_uuid_or_404`). Hardening, not a fix on its own.
3. A reconciliation pass to drop `identity_override:*` documents whose `identity_uuid` no longer resolves. Existing orphans cannot be removed through the product, so they need clearing separately.
Contributor guide
Research direction
Start with EdgeIdentity.save(), _get_changes, and _initial_state to trace identity writes and the environments_v2 diff. Compare the conditional-write approach in DynamoIdentityWrapper.set_system_trait and inspect get_item_from_uuid_or_404 for the UUID lookup. Done means concurrent identity deletion and override writes no longer leave orphaned identity_override documents, with existing orphans handled by reconciliation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100