elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Decouple core eval loop from Kibana

Open
#44 0 comments 0 reactions 1 assignee Claimed by @milistu View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.

### Proposed approach

Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.

Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.

Agree on the port interfaces before starting: this touches the core runner.

### Open questions

- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?

### Done when

- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.