mongodb-developer / mongodb-developer/GenAI-Showcase
Notebook proposal: EvalPort import/export alongside the existing notebooks/evals/ examples
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.3k
- Forks
- 746
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 2
Description
Hi — I maintain EvalPort (Apache 2.0), an open JSON spec for portable LLM evaluation datasets (test cases, graders, suites, results) designed to move between eval frameworks without losing meaning.
I looked at notebooks/evals/ before writing this — you already have ragas-evaluation.ipynb, the embeddings-eval notebooks (openai-embeddings-eval.ipynb, voyageai-embeddings-eval.ipynb, angle-embeddings-eval.ipynb, gemini-embeddings-eval.ipynb), and Patronus_MongoDB.ipynb, all indexed in that folder's README.md. That's a genuinely good fit for a small companion notebook: take the query/ground-truth pairs one of those notebooks already evaluates (e.g. the Ragas one) and show them exported as a portable EvalPort suite + ResultSet, so the same eval data is usable outside Ragas too.
Rough shape:
from openeval.validate import validate_suite, validate_result_set
suite = {
"version": "1.0.0",
"id": "mongodb-rag-eval",
"graders": [{"id": "gr1", "type": "exact_match"}],
"test_cases": [
{"id": "tc1", "input": "...", "expected_output": "...", "graders": ["gr1"]}
],
}
assert validate_suite(suite).valid
# ... run against the MongoDB Atlas-backed pipeline, grade, build a ResultSet ...
Would a notebook like this be a welcome addition to notebooks/evals/, listed alongside the others in that folder's README? Keeping the ask scoped to one self-contained notebook per your CONTRIBUTING.md norms — no pressure if this isn't the right fit right now.
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with notebooks/evals/ragas-evaluation.ipynb and notebooks/evals/README.md, then review CONTRIBUTING.md and EvalPort's SPEC.md. Add one self-contained notebook that exports the existing evaluation pairs as an EvalPort suite and ResultSet, validates them, and is listed in the README.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, mongodb, python
- Domain
- databases, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100