Flagsmith / Flagsmith/flagsmith
Spike: SDK API E2E test suite
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 567
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 121
Description
## Context
Following @Zaimwa9's [E2E test improvements](https://github.com/Flagsmith/edge-api/pull/535) for Edge API ([edge-api#528](https://github.com/Flagsmith/edge-api/issues/528)), we have a solid set of SDK API E2E tests covering `/flags`, `/identities`, and `/environment-document`. However:
1. Test setup is manual. The edge-api tests are tightly coupled to a pre-configured "Edge API E2E" Flagsmith project on staging/prod. Test data must be created by hand in the dashboard. This either makes the test data append only, or tests fragile and hard to modify.
1. Tests only run against Edge API. The Core API has in-process integration tests but no true E2E tests hitting a running server over HTTP for the SDK endpoints. Ideally, the SDK API contract should be verified identically for both Core and Edge.
## Goal
Self-contained SDK API E2E tests that set up their own test data via the management API, requiring only a base URL and a Master API key as input. The same tests should run against both Core API and Edge API, and failures should block PRs in Core.
The outcome of this spike should be either a wontfix, or an epic detailing out the implementation phases.
## Proposed phases
### Phase 1: Have a management API client for test setup
Generate a typed Python client from our management API OpenAPI spec (`/api/v1/swagger.json` via drf-spectacular) to programmatically create all test data: projects, environments, features, segments with rules, feature states, segment overrides, identities, traits, and server-side API keys.
This could be expanded into its own task (i.e. providing a Stainless-managed client for public use), or omitted in favour of a hand-written client.
**Deliverable:** A management API client package usable as a test dependency.
### Phase 2: Translate existing test projects to setup code
The "Edge API E2E" projects currently configured on staging and prod contain the test data that the E2E tests rely on (features, segments, MV variants, identities with traits, etc.). This phase translates that manual configuration into programmatic setup using the generated management API client from Phase 1.
The setup should be self-contained: given a `management_api_url` and a `master_api_key`, it recreates the full test project from scratch and returns the `client_key` and `server_key` needed by the tests. Teardown deletes the project (cascading). MV percentage allocations should be pinned for deterministic testing.
**Key design point:** `management_api_url` and `sdk_api_url` can differ. For Core API, both point at the same server. For Edge API, setup goes through Core's management API while SDK tests hit the Edge API URL.
### Phase 3: Share tests between Core API and Edge API
Make the SDK API E2E test suite consumable by both repos. Options:
- `flagsmith-common`
- `flagsmith/flagsmith` — e.g. alongside OpenAPI spec in `sdk/`
- Code duplication between `flagsmith/flagsmith` and `flagsmith/edge-api`
The test suite should be parameterisable:
- `management_api_url` — for setup (always Core API)
- `master_api_key` — for setup authentication
- `sdk_api_url` — for the actual SDK tests
### Phase 4: Integrate into Core API CI
The SDK API E2E tests should run **alongside the existing frontend E2E tests**, using the same Flagsmith instances and triggers, and should be PR-blocking.
Add a CI step in `.reusable-docker-e2e-tests.yml` that runs the SDK API tests after the Docker Compose stack is up (before or after the TestCafe `make test` step). The tests target the same `flagsmith-api` container already running in the stack. They set up their own project via the management API, run, and tear down — no interference with the frontend E2E seed data.
**Master API key:** Either block on #6797 (unified provisioning system) which would provide a declarative way to provision a Master API key during instance bootstrap, or extend `seed_data()` to create one as a shorter-term solution.
**Live API mode (staging / production):** Add a step to `frontend-deploy-production.yml` and `frontend-test-staging.yml` to also run SDK API tests, using a Master API key stored as a GitHub environment secret.
**Edge API:** The same test suite runs in `flagsmith/edge-api` CI, with `management_api_url` pointing to the Core API for setup and `sdk_api_url` pointing to the Edge API under test.
### Nice-to-haves (not blocking)
- **Public management API client** — if we go with Stainless, the generated client could be useful for customers automating their Flagsmith setup
## Related
- #6797 potential dependency for Master API key bootstrapping
- #5397, #6739 pointing out differing behaviour across Core/Edge
- [engine-test-data#29](https://github.com/Flagsmith/engine-test-data/issues/29) Duplicate?
- [edge-api#528](https://github.com/Flagsmith/edge-api/issues/528) Improve e2e tests coverage (closed)
- [edge-api#535](https://github.com/Flagsmith/edge-api/pull/535) feat/improve edge api e2e tests (merged)
## Open questions
- [ ] Should we implement running SDK API E2E against Edge PRs as part of this task?
- [ ] Where should the shared test package live? (`flagsmith-common`, standalone, or in this repo under `sdk/`)
- [ ] Should we evaluate Stainless for a public management API client, or keep this purely internal?
Contributor guide
Assessment
This issue has not been assessed yet.