elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Experiment aborts on any exception
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
The executor has no error handling around evaluator calls. One network blip during a long run kills everything. Already-ingested scores survive, but the rest of the run is lost with no partial result.
### Problem
`executor/client.py:174-180` calls `await evaluator.evaluate(params)` with no try/except. `asyncio.gather` without `return_exceptions=True` means one failure propagates and aborts the whole run.
Kibana evaluators propagate exceptions. The executor has no safety net.
### Fix
Wrap evaluator calls in the executor. Catch exceptions, convert to `EvaluationResult(label="error", explanation=...)`, log at error level, and continue the run.
kbn-evals wraps every evaluator call in try/catch and converts exceptions to a `{status: "error"}` entry without aborting the run. The SDK should mirror that.
### Done when
- [ ] A failed evaluator does not abort other evaluators or the run
- [ ] The failed score is ingested with `label="error"` and an explanation
- [ ] The test `test_kibana_evaluators_propagate_request_errors` is updated to match the new policy
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in executor/client.py:174-180, where evaluator.evaluate(params) is called, and run test_kibana_evaluators_propagate_request_errors. Trace the asyncio.gather path and existing EvaluationResult handling. Done means evaluator failures are logged, ingested as label="error" with an explanation, and do not abort other evaluators or the run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100