elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Decouple core eval loop from Kibana

Chiusa
#44 0 commenti 0 reazioni 1 assegnatario Rivendicata da @milistu Vedi su GitHub
kbn-evals Team:nightshift-context-and-research
Lingua principale
Python
Stelle
2
Fork
0
Merge medio
1g 13h
PR unite (30g)
18

Descrizione

### Summary

`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.

### Proposed approach

Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.

Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.

Agree on the port interfaces before starting: this touches the core runner.

### Open questions

- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?

### Done when

- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.