open-feature / open-feature/flagd-testbed

Migrate SDKs to centralized evaluator testkit and remove duplicate evaluation tests

Open
#366 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Gherkin
Stars
5
Forks
14
Avg merge
3h 17m
Merged PRs (30d)
13

Description

Overview

PR #344 introduced the flagd-api-testkit with a dedicated evaluator test suite in evaluator/gherkin/.
This issue tracks the migration of all SDKs to use this new testkit and eliminates duplicated evaluation
tests currently scattered across individual SDK repositories.

Current State

  • ✅ Evaluator testkit created (see PR #344)
  • ❌ SDKs still running duplicate evaluation tests locally
  • ❌ Inconsistent test organization across SDKs

Goals

  • Migrate all SDKs to use the centralized evaluator testkit from flagd-testbed
  • Remove duplicate evaluation test definitions from each SDK
  • Keep only SDK-specific integration/provider tests in each repository
  • Establish clear boundaries: evaluation tests (testkit) vs. integration tests (SDK repo)

Required Changes Per SDK

For each SDK (Java, Go, JS, .NET, Python, etc.):

  • Remove duplicate Gherkin test files related to evaluation
  • Remove unnecessary step definitions for evaluation scenarios
  • Update test runner to pull evaluator tests from flagd-testbed via git submodule
  • Update CI/CD to run both evaluator testkit and integration-only tests
  • Document the new test structure in SDK README

Example Structure

SDK Repository
├── tests/integration/           # Only provider/connection/config tests
│   └── gherkin/
├── flagd-testbed/              # Git submodule
│   └── evaluator/gherkin/      # Shared evaluator tests (via testkit)

Benefits

  • ✅ Single source of truth for evaluator tests
  • ✅ Faster CI/CD (evaluator tests run without Docker where possible)
  • ✅ Reduced maintenance burden
  • ✅ Consistent evaluation behavior across all SDKs

Out of Scope (for this issue)

  • Creating new features or test scenarios
  • Changes to the evaluator testkit itself (separate issues/PRs)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PR #344 and the shared evaluator suite in evaluator/gherkin/, then compare each SDK's duplicate Gherkin tests, step definitions, runners, CI configuration, and README. The work is complete when SDK repositories retain only integration/provider tests, consume flagd-testbed through the submodule, run both test suites in CI, and document the new structure.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
ci-cd, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.