elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Decouple core eval loop from Kibana
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.
### Proposed approach
Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.
Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.
Agree on the port interfaces before starting: this touches the core runner.
### Open questions
- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?
### Done when
- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.