elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Add IR evaluators with rank-based metrics (HitRate@K, MRR@K, NDCG@K, MAP@K)

Open
#60 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

The SDK has no information retrieval evaluators in the installed package. `create_document_recall_evaluator` in `examples/` covers plain recall only and is not exported.

`@kbn/evals` (TypeScript) ships seven IR metrics under `create_ir_evaluators`. Reference PR: elastic/kibana#288639 (waiting for approval). The Python SDK needs a matching implementation for local CODE evaluators.

### Proposed approach

Add `src/elastic_evals/evaluators/ir/` with:

- `RetrievedDoc`: `{ index: str, id: str }`: rank implied by list order
- `GroundTruth`: `dict[str, dict[str, float]]`: `{ index_name: { doc_id: relevance_score } }`
- `create_ir_evaluators(config)` → `list[Evaluator]`: all seven metrics, one set per K value
- Per-metric factories: `create_precision_at_k_evaluator`, `create_recall_at_k_evaluator`, etc.
- `IR_EVAL_K` env var (comma-separated) to override configured K at runtime

Metric formulas mirror elastic/kibana#288639 exactly:
- NDCG: graded relevance, linear DCG $Σ gain_i / log2(i + 2)$, ideal gains from full ground truth
- MAP denominator: `min(K, total_relevant)`
- Deduplicate retrieved docs before top-K cutoff

Export all public symbols from `elastic_evals.evaluators`.

### Done when

- [ ] All 7 metrics implemented and formula-verified against elastic/kibana#288639
- [ ] `create_ir_evaluators` exported from `elastic_evals.evaluators`
- [ ] `IR_EVAL_K` env var supported
- [ ] Unit tests cover all metrics and edge cases (no ground truth, duplicates, multi-K)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.