elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Evaluator discovery and preflight validation

Open
#33 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

A typo in an evaluator name (`KibanaEvaluatorConfig(name="corectness")`) only surfaces at scoring time as `label="error"`, after the task has already run. Two methods for discovery and validation already exist in the API client but are wired to nothing.

### Problem

- `api/evaluators_client.py:48`: `list_evaluators()` exists but is not called in production code
- `api/evaluators_client.py:62`: `validate()` exists but is not called in production code
- `KibanaEvaluatorConfig.name` is a plain `str` with no validation; even an empty string is accepted
- No CLI command exists to list Kibana evaluators (a command does exist to list suites)

### Fix

- Validate evaluator names before the run starts using `list_evaluators()` or `validate()`
- Add an `elastic-evals list-evaluators` CLI command
- Wire this into the named classmethod approach (see #36)

### Done when

- [ ] A misspelt evaluator name raises a clear error before any task runs
- [ ] `elastic-evals list-evaluators` prints the evaluators available in Kibana

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.