elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Decouple core eval loop from Kibana
- Lingua principale
- Python
- Stelle
- 2
- Fork
- 0
- Merge medio
- 1g 13h
- PR unite (30g)
- 18
Descrizione
### Summary
`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.
### Proposed approach
Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.
Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.
Agree on the port interfaces before starting: this touches the core runner.
### Open questions
- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?
### Done when
- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.