open-feature / open-feature/flagd-testbed
Migrate SDKs to centralized evaluator testkit and remove duplicate evaluation tests
Nobody has claimed this yet.
- Dominant language
- Gherkin
- Stars
- 5
- Forks
- 14
- Avg merge
- 3h 17m
- Merged PRs (30d)
- 13
Description
Overview
PR #344 introduced the flagd-api-testkit with a dedicated evaluator test suite in evaluator/gherkin/.
This issue tracks the migration of all SDKs to use this new testkit and eliminates duplicated evaluation
tests currently scattered across individual SDK repositories.
Current State
- ✅ Evaluator testkit created (see PR #344)
- ❌ SDKs still running duplicate evaluation tests locally
- ❌ Inconsistent test organization across SDKs
Goals
- Migrate all SDKs to use the centralized evaluator testkit from flagd-testbed
- Remove duplicate evaluation test definitions from each SDK
- Keep only SDK-specific integration/provider tests in each repository
- Establish clear boundaries: evaluation tests (testkit) vs. integration tests (SDK repo)
Required Changes Per SDK
For each SDK (Java, Go, JS, .NET, Python, etc.):
- Remove duplicate Gherkin test files related to evaluation
- Remove unnecessary step definitions for evaluation scenarios
- Update test runner to pull evaluator tests from flagd-testbed via git submodule
- Update CI/CD to run both evaluator testkit and integration-only tests
- Document the new test structure in SDK README
Example Structure
SDK Repository
├── tests/integration/ # Only provider/connection/config tests
│ └── gherkin/
├── flagd-testbed/ # Git submodule
│ └── evaluator/gherkin/ # Shared evaluator tests (via testkit)
Benefits
- ✅ Single source of truth for evaluator tests
- ✅ Faster CI/CD (evaluator tests run without Docker where possible)
- ✅ Reduced maintenance burden
- ✅ Consistent evaluation behavior across all SDKs
Out of Scope (for this issue)
- Creating new features or test scenarios
- Changes to the evaluator testkit itself (separate issues/PRs)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with PR #344 and the shared evaluator suite in evaluator/gherkin/, then compare each SDK's duplicate Gherkin tests, step definitions, runners, CI configuration, and README. The work is complete when SDK repositories retain only integration/provider tests, consume flagd-testbed through the submodule, run both test suites in CI, and document the new structure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git
- Domain
- ci-cd, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100