elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Non-dict expected output is silently dropped before Kibana upload
- Lingua principale
- Python
- Stelle
- 2
- Fork
- 0
- Merge medio
- 1g 13h
- PR unite (30g)
- 18
Descrizione
**Describe the bug**
A non-dict expected `output` on an `Example` (e.g. `output="4"`) is silently replaced with `None` by `_dict_or_none` in `KibanaDatasetStore` before upload. The upload succeeds, evaluators receive `expected=None`, and the run produces wrong scores with no error.
Kibana would have rejected the request with a 400: its upsert schema requires `output` to be a JSON object (`kbn-evals-common/impl/schemas/datasets/upsert_dataset_route.gen.ts`).
Only `output` is affected. `TInput` and `TMetadata` in `elastic_evals.types` are bounded to `dict`, so pydantic already rejects non-dict values there. `TExpected` has no bound.
**To reproduce**
1. `Example(input={"q": "2+2"}, output="4")`
2. Run against Kibana with an exact-match evaluator.
3. Every example scores 0; nothing is logged.
**Expected behavior**
Constructing the `Example` fails with a clear validation error.
**Proposed fix**
Bound `TExpected` to `dict[str, Any]`, matching `TInput` and `TMetadata`. One rule, enforced at construction, for both local and Kibana runs. Then:
- `examples/claude_code_eval/datasets/coding_tasks.py` uses `None` as the expected type argument; change to `dict[str, Any]`.
- Remove the now-dead `_dict_or_none` calls in `KibanaDatasetStore`.
- Add a test that `Example(input={}, output="x")` raises.
**Alternative considered**
Raise only in `KibanaDatasetStore`, keeping `output` free for in-memory runs. Rejected: local and Kibana runs would accept different datasets.
**Additional context**
Predates #65, which only moved the helper from the runner into `KibanaDatasetStore`. Found while working on #44.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia leggendo le definizioni del tipo Example in elastic_evals.types e KibanaDatasetStore, quindi esamina examples/claude_code_eval/datasets/coding_tasks.py. Esegui la suite di test esistente prima di apportare modifiche. Il lavoro è completo quando un output di Example che non è un dizionario genera un errore di validazione chiaro, il tipo coding-tasks viene aggiornato e le chiamate inutilizzate a _dict_or_none vengono rimosse.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- backend-api-design, testing
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Attiva
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 78/100