open-feature / open-feature/flagd-testbed
evaluator: fractional-nested-var tests should assert error-code for invalid variant resolution
Nobody has claimed this yet.
- Dominant language
- Gherkin
- Stars
- 5
- Forks
- 14
- Avg merge
- 3h 17m
- Merged PRs (30d)
- 13
Description
Context
The fractional-nested-var test scenarios in evaluator/gherkin/fractional.feature test the case where var("color") resolves to a value (e.g., "yellow" or "") that is not a valid variant name. The current test only asserts the resolved value equals the fallback:
| jon@company.com | yellow | fallback |
| jon@company.com | | fallback |
Problem
Every evaluator implementation returns a GENERAL error when the resolved variant doesn't exist in the flag's variants map. The test step definitions in each SDK testkit work around this by catching the error and substituting the fallback value (mimicking SDK client behavior):
Python (evaluation_steps.py):
except OpenFeatureError as e:
return FlagResolutionDetails(default_value, error_code=e.error_code, reason=Reason.ERROR)
Java (EvaluationSteps.java):
catch (OpenFeatureError e) {
state.evaluation = ProviderEvaluation.builder()
.value(state.defaultValue)
.errorCode(e.getErrorCode())
.build();
}
Go/flagd (gherkin_test.go):
if errCode != model.FlagNotFoundErrorCode && errCode != model.TypeMismatchErrorCode {
tc.resultValue = parseTypedValue(tc.flagType, tc.defaultVal)
tc.resultReason = model.DefaultReason
tc.resultError = nil
}
All three implementations converge on the same workaround: catch the error, return the fallback value, and effectively hide the error from the assertion.
Proposal
The Gherkin tests should make the expected behavior explicit. Options:
- Assert the error code: Add
And the error-code should be "GENERAL"to make it clear an error is expected - Assert the reason: Add
And the reason should be "ERROR" - Define evaluator-level fallback behavior: If the evaluator should return the fallback value (not an error), document this and update evaluator implementations accordingly
This would eliminate the need for SDK-level workarounds in all testkits and make the expected behavior unambiguous.
Affected implementations
- Python: open-feature/python-sdk-contrib#377
- Java:
tools/flagd-api-testkitin java-sdk-contrib (on main) - Go/flagd: open-feature/flagd#1947
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with evaluator/gherkin/fractional.feature and inspect the fractional-nested-var scenarios and available error-code and reason assertion steps. Compare them with the Python, Java, and Go/flagd step definitions, then establish the expected contract and update the scenarios so completion is demonstrated by an explicit error-code or reason assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100