elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Decouple core eval loop from Kibana

Geschlossen
#44 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @milistu Auf GitHub ansehen
kbn-evals Team:nightshift-context-and-research
Vorherrschende Sprache
Python
Sterne
2
Forks
0
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
18

Beschreibung

### Summary

`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.

### Proposed approach

Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.

Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.

Agree on the port interfaces before starting: this touches the core runner.

### Open questions

- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?

### Done when

- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.