elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Non-dict expected output is silently dropped before Kibana upload

Open
#70 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

**Describe the bug**

A non-dict expected `output` on an `Example` (e.g. `output="4"`) is silently replaced with `None` by `_dict_or_none` in `KibanaDatasetStore` before upload. The upload succeeds, evaluators receive `expected=None`, and the run produces wrong scores with no error.

Kibana would have rejected the request with a 400: its upsert schema requires `output` to be a JSON object (`kbn-evals-common/impl/schemas/datasets/upsert_dataset_route.gen.ts`).

Only `output` is affected. `TInput` and `TMetadata` in `elastic_evals.types` are bounded to `dict`, so pydantic already rejects non-dict values there. `TExpected` has no bound.

**To reproduce**

1. `Example(input={"q": "2+2"}, output="4")`
2. Run against Kibana with an exact-match evaluator.
3. Every example scores 0; nothing is logged.

**Expected behavior**

Constructing the `Example` fails with a clear validation error.

**Proposed fix**

Bound `TExpected` to `dict[str, Any]`, matching `TInput` and `TMetadata`. One rule, enforced at construction, for both local and Kibana runs. Then:

- `examples/claude_code_eval/datasets/coding_tasks.py` uses `None` as the expected type argument; change to `dict[str, Any]`.
- Remove the now-dead `_dict_or_none` calls in `KibanaDatasetStore`.
- Add a test that `Example(input={}, output="x")` raises.

**Alternative considered**

Raise only in `KibanaDatasetStore`, keeping `output` free for in-memory runs. Rejected: local and Kibana runs would accept different datasets.

**Additional context**

Predates #65, which only moved the helper from the runner into `KibanaDatasetStore`. Found while working on #44.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.