elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Replace evaluator factories with named classmethods

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

The factory functions have two problems. First, combining two factories of the same family (e.g. analysis + quantitative) silently doubles the Kibana judge API calls per trace because each factory creates its own `_EvaluationBatch`. Second, the four CODE evaluator factories are byte-for-byte identical except for three strings (docstring, function name, and the `name=` argument), split across eight files.

### Proposed API

```python
kibana_evaluators([
KibanaEvaluatorConfig.correctness(connector_id="..."),
KibanaEvaluatorConfig.groundedness(connector_id="..."),
KibanaEvaluatorConfig.latency(),
KibanaEvaluatorConfig.input_tokens(),
], client=client)
```

One `kibana_evaluators()` call = one HTTP request per trace, regardless of how many evaluators are in the list. IDE autocomplete on `KibanaEvaluatorConfig.` shows everything available. The double-cost problem is impossible by design.

### Done when

- [ ] `KibanaEvaluatorConfig` has named classmethods for all built-in evaluator types
- [ ] The standalone `create_*` factory functions are removed
- [ ] README and evaluator reference show the composable API as the primary approach
- [ ] One `kibana_evaluators()` call with multiple configs produces one HTTP request per trace

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.