elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Add IR evaluators with rank-based metrics (HitRate@K, MRR@K, NDCG@K, MAP@K)
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
The SDK has no information retrieval evaluators in the installed package. `create_document_recall_evaluator` in `examples/` covers plain recall only and is not exported.
`@kbn/evals` (TypeScript) ships seven IR metrics under `create_ir_evaluators`. Reference PR: elastic/kibana#288639 (waiting for approval). The Python SDK needs a matching implementation for local CODE evaluators.
### Proposed approach
Add `src/elastic_evals/evaluators/ir/` with:
- `RetrievedDoc`: `{ index: str, id: str }`: rank implied by list order
- `GroundTruth`: `dict[str, dict[str, float]]`: `{ index_name: { doc_id: relevance_score } }`
- `create_ir_evaluators(config)` → `list[Evaluator]`: all seven metrics, one set per K value
- Per-metric factories: `create_precision_at_k_evaluator`, `create_recall_at_k_evaluator`, etc.
- `IR_EVAL_K` env var (comma-separated) to override configured K at runtime
Metric formulas mirror elastic/kibana#288639 exactly:
- NDCG: graded relevance, linear DCG $Σ gain_i / log2(i + 2)$, ideal gains from full ground truth
- MAP denominator: `min(K, total_relevant)`
- Deduplicate retrieved docs before top-K cutoff
Export all public symbols from `elastic_evals.evaluators`.
### Done when
- [ ] All 7 metrics implemented and formula-verified against elastic/kibana#288639
- [ ] `create_ir_evaluators` exported from `elastic_evals.evaluators`
- [ ] `IR_EVAL_K` env var supported
- [ ] Unit tests cover all metrics and edge cases (no ground truth, duplicates, multi-K)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.