elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Non-dict expected output is silently dropped before Kibana upload
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
**Describe the bug**
A non-dict expected `output` on an `Example` (e.g. `output="4"`) is silently replaced with `None` by `_dict_or_none` in `KibanaDatasetStore` before upload. The upload succeeds, evaluators receive `expected=None`, and the run produces wrong scores with no error.
Kibana would have rejected the request with a 400: its upsert schema requires `output` to be a JSON object (`kbn-evals-common/impl/schemas/datasets/upsert_dataset_route.gen.ts`).
Only `output` is affected. `TInput` and `TMetadata` in `elastic_evals.types` are bounded to `dict`, so pydantic already rejects non-dict values there. `TExpected` has no bound.
**To reproduce**
1. `Example(input={"q": "2+2"}, output="4")`
2. Run against Kibana with an exact-match evaluator.
3. Every example scores 0; nothing is logged.
**Expected behavior**
Constructing the `Example` fails with a clear validation error.
**Proposed fix**
Bound `TExpected` to `dict[str, Any]`, matching `TInput` and `TMetadata`. One rule, enforced at construction, for both local and Kibana runs. Then:
- `examples/claude_code_eval/datasets/coding_tasks.py` uses `None` as the expected type argument; change to `dict[str, Any]`.
- Remove the now-dead `_dict_or_none` calls in `KibanaDatasetStore`.
- Add a test that `Example(input={}, output="x")` raises.
**Alternative considered**
Raise only in `KibanaDatasetStore`, keeping `output` free for in-memory runs. Rejected: local and Kibana runs would accept different datasets.
**Additional context**
Predates #65, which only moved the helper from the runner into `KibanaDatasetStore`. Found while working on #44.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.