elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Small correctness (UX) fixes
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
Seven small fixes, each a few lines.
1. `run_experiment()` has no summary output. Add a `verbose: bool = True` parameter that prints a result table at the end using `rich`. (`executor/client.py`)
2. 207 partial-ingest failures are logged but the run proceeds as if everything worked. The caller should know which scores failed. (`scores_client.py:45-54`)
3. `task_output.pop("_interaction_trace_id", ...)` mutates the dict returned by the task. Take a shallow copy first. (`client.py:152`)
4. `evaluation_runs[-1]` in `client.py:208` assumes the last item in the list belongs to the current example. No `await` exists between the append and this read today, but a single `await` added later would silently corrupt results. Capture the object directly at append time instead.
5. `validate_log_level` runs after Pydantic's `Literal` coercion so the check always passes. It's dead code. Remove it. (`config.py`)
6. CI build provenance is hardcoded to `None`. Read the standard Buildkite env vars (`BUILDKITE_BUILD_ID`, `BUILDKITE_JOB_ID`, etc.) when present. (`client.py:111`)
7. `except httpx.HTTPError: raise` in `scores_client.py:42-43` catches an exception and immediately re-raises it unchanged. Remove it.
### Done when
- [ ] All seven items above are fixed
- [ ] Existing tests pass
Contributor guide
No contributing guide indexed for this repository
Research direction
Read executor/client.py, scores_client.py, client.py, and config.py first, then inspect the existing tests covering these clients and configuration paths. Implement all seven listed correctness and cleanup changes, including the requested output and Buildkite metadata behavior, and run the existing test suite to confirm it passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100