elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Experiment aborts on any exception

Open Beginner friendly
#31 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

The executor has no error handling around evaluator calls. One network blip during a long run kills everything. Already-ingested scores survive, but the rest of the run is lost with no partial result.

### Problem

`executor/client.py:174-180` calls `await evaluator.evaluate(params)` with no try/except. `asyncio.gather` without `return_exceptions=True` means one failure propagates and aborts the whole run.

Kibana evaluators propagate exceptions. The executor has no safety net.

### Fix

Wrap evaluator calls in the executor. Catch exceptions, convert to `EvaluationResult(label="error", explanation=...)`, log at error level, and continue the run.

kbn-evals wraps every evaluator call in try/catch and converts exceptions to a `{status: "error"}` entry without aborting the run. The SDK should mirror that.

### Done when

- [ ] A failed evaluator does not abort other evaluators or the run
- [ ] The failed score is ingested with `label="error"` and an explanation
- [ ] The test `test_kibana_evaluators_propagate_request_errors` is updated to match the new policy

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in executor/client.py:174-180, where evaluator.evaluate(params) is called, and run test_kibana_evaluators_propagate_request_errors. Trace the asyncio.gather path and existing EvaluationResult handling. Done means evaluator failures are logged, ingested as label="error" with an explanation, and do not abort other evaluators or the run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.