Flagsmith / Flagsmith/flagsmith

Enabled features can disagree with the flags actually served after a scheduled change

Open
#8,127 0 comments 0 reactions 1 assignee Claimed by @emyller View on GitHub
Dominant language
Python
Stars
6.6k
Forks
567
Avg merge
1d 13h
Merged PRs (30d)
121

Description

When a scheduled change request and a direct edit to the same flag overlap, parts of Flagsmith stop agreeing with what the SDKs actually serve.

To reproduce, in an environment on v1 versioning:

1. Create a change request that turns a flag off, scheduled for tomorrow.
2. Before it goes live, edit the same flag directly and leave it on.
3. Wait for the scheduled time to pass.

The SDKs now serves the flag as off, which is expected as the scheduled change is the most recent one to go live. But the dashboard still counts the flag as enabled.

The reason is that the flag evaluation code picks the version that went live most recently, while several other places pick the version that was *created* most recently. A change request is created before the direct edit but goes live after it, so the two disagree. Anywhere a flag has more than one version, the second kind of code can show you a superseded value.

What this means for the user, by feature:

| What you see | What can go wrong |
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Flags served by SDKs | Correct. This is the behaviour everything else should match. |
| Environment metrics: enabled features count | Counts the flag above as enabled when it is being served as off. |
| Experiment health: sample ratio mismatch warning | Can warn about a traffic split you have already replaced, or miss a real mismatch. |
| Importing flags into an environment | The import fails outright once the flag you are importing into has any version history. |
| Audit log entries for new segment and identity overrides | A new override can be missing from the audit log, or logged twice, after a scheduled change. |
| GitHub issue and PR comments on linked features | The comment can list old flag values alongside the current one. |
| Multivariate options endpoint used by Vercel Flags Explorer | Serves an arbitrary version once the flag has history. |
| GitLab merge request comments showing per-environment flag states | Shows an arbitrary version once the flag has history. |
| Experiment variant stickiness | No impact — this deliberately looks at history and must keep doing so (#7913). |
| Environment metrics: identity and segment override counts | No impact — these count how many overrides exist, not their values. |
| Experiment rollout split | No impact in practice. |

Technical detail

The metrics query picks the newest feature state by `Max(id)` (`api/environments/models.py:432`), while evaluation orders by (`live_from`, `version`). Every consumer of `get_live_feature_states` that picks a single winner by anything other than go-live order is affected; the import path assumes a single live row and calls `.get()`, hence `MultipleObjectsReturned`.

Proposed fix:

- Add a `get_latest_live_feature_states` function, encapsulating the SQL from #8096.
- Migrate the affected callers, each with a test pinning the new behaviour. The sample ratio mismatch code notes that it matches the environment metrics convention, so those two move together.
- Keep `FeatureState.get_superseded_live_feature_state` on the current `get_live_feature_states` implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.