elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Replace evaluator factories with named classmethods
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
The factory functions have two problems. First, combining two factories of the same family (e.g. analysis + quantitative) silently doubles the Kibana judge API calls per trace because each factory creates its own `_EvaluationBatch`. Second, the four CODE evaluator factories are byte-for-byte identical except for three strings (docstring, function name, and the `name=` argument), split across eight files.
### Proposed API
```python
kibana_evaluators([
KibanaEvaluatorConfig.correctness(connector_id="..."),
KibanaEvaluatorConfig.groundedness(connector_id="..."),
KibanaEvaluatorConfig.latency(),
KibanaEvaluatorConfig.input_tokens(),
], client=client)
```
One `kibana_evaluators()` call = one HTTP request per trace, regardless of how many evaluators are in the list. IDE autocomplete on `KibanaEvaluatorConfig.` shows everything available. The double-cost problem is impossible by design.
### Done when
- [ ] `KibanaEvaluatorConfig` has named classmethods for all built-in evaluator types
- [ ] The standalone `create_*` factory functions are removed
- [ ] README and evaluator reference show the composable API as the primary approach
- [ ] One `kibana_evaluators()` call with multiple configs produces one HTTP request per trace
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.