elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Small correctness (UX) fixes

Open
#42 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

Seven small fixes, each a few lines.

1. `run_experiment()` has no summary output. Add a `verbose: bool = True` parameter that prints a result table at the end using `rich`. (`executor/client.py`)
2. 207 partial-ingest failures are logged but the run proceeds as if everything worked. The caller should know which scores failed. (`scores_client.py:45-54`)
3. `task_output.pop("_interaction_trace_id", ...)` mutates the dict returned by the task. Take a shallow copy first. (`client.py:152`)
4. `evaluation_runs[-1]` in `client.py:208` assumes the last item in the list belongs to the current example. No `await` exists between the append and this read today, but a single `await` added later would silently corrupt results. Capture the object directly at append time instead.
5. `validate_log_level` runs after Pydantic's `Literal` coercion so the check always passes. It's dead code. Remove it. (`config.py`)
6. CI build provenance is hardcoded to `None`. Read the standard Buildkite env vars (`BUILDKITE_BUILD_ID`, `BUILDKITE_JOB_ID`, etc.) when present. (`client.py:111`)
7. `except httpx.HTTPError: raise` in `scores_client.py:42-43` catches an exception and immediately re-raises it unchanged. Remove it.

### Done when

- [ ] All seven items above are fixed
- [ ] Existing tests pass

Contributor guide

No contributing guide indexed for this repository

Research direction

Read executor/client.py, scores_client.py, client.py, and config.py first, then inspect the existing tests covering these clients and configuration paths. Implement all seven listed correctness and cleanup changes, including the requested output and Buildkite metadata behavior, and run the existing test suite to confirm it passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.