elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Decouple core eval loop from Kibana
- Vorherrschende Sprache
- Python
- Sterne
- 2
- Forks
- 0
- Ø Merge
- 1 T. 13 Std.
- Gemergte PRs (30 T.)
- 18
Beschreibung
### Summary
`ElasticEvalsClient.__init__` builds three Kibana clients unconditionally. `connector_id` is always required even for runs that don't use inference, and you can't unit-test the core loop without mocking HTTP. There is no local-only mode and no way to run against an existing dataset without replacing it first.
### Proposed approach
Inject a `DatasetStore` and `ScoreSink` into the constructor with Kibana implementations as defaults. This extends the lazy-factory approach of `get_inference_client()` to the dataset store and score sink, making them replaceable at construction time.
Keep score ingestion incremental (per-example, not post-loop). Post-loop ingestion would lose all scores on a crash. kbn-evals ingests incrementally by design.
Agree on the port interfaces before starting: this touches the core runner.
### Open questions
- Should a local run skip Kibana score ingestion entirely, or can the API accept locally generated IDs?
- What should happen when local and remote examples differ?
### Done when
- [ ] `ElasticEvalsClient` can be constructed without Kibana credentials when alternative store/sink implementations are provided
- [ ] The core loop can be unit-tested without HTTP mocks
- [ ] A local mode exists: examples run without uploading to Kibana first
- [ ] Score ingestion stays incremental
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.