mongodb-developer / mongodb-developer/GenAI-Showcase

Notebook proposal: EvalPort import/export alongside the existing notebooks/evals/ examples

Open
#227 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.3k
Forks
746
Avg merge
1d 7h
Merged PRs (30d)
2

Description

Hi — I maintain EvalPort (Apache 2.0), an open JSON spec for portable LLM evaluation datasets (test cases, graders, suites, results) designed to move between eval frameworks without losing meaning.

I looked at notebooks/evals/ before writing this — you already have ragas-evaluation.ipynb, the embeddings-eval notebooks (openai-embeddings-eval.ipynb, voyageai-embeddings-eval.ipynb, angle-embeddings-eval.ipynb, gemini-embeddings-eval.ipynb), and Patronus_MongoDB.ipynb, all indexed in that folder's README.md. That's a genuinely good fit for a small companion notebook: take the query/ground-truth pairs one of those notebooks already evaluates (e.g. the Ragas one) and show them exported as a portable EvalPort suite + ResultSet, so the same eval data is usable outside Ragas too.

Rough shape:

from openeval.validate import validate_suite, validate_result_set

suite = {
    "version": "1.0.0",
    "id": "mongodb-rag-eval",
    "graders": [{"id": "gr1", "type": "exact_match"}],
    "test_cases": [
        {"id": "tc1", "input": "...", "expected_output": "...", "graders": ["gr1"]}
    ],
}
assert validate_suite(suite).valid
# ... run against the MongoDB Atlas-backed pipeline, grade, build a ResultSet ...

Would a notebook like this be a welcome addition to notebooks/evals/, listed alongside the others in that folder's README? Keeping the ask scoped to one self-contained notebook per your CONTRIBUTING.md norms — no pressure if this isn't the right fit right now.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with notebooks/evals/ragas-evaluation.ipynb and notebooks/evals/README.md, then review CONTRIBUTING.md and EvalPort's SPEC.md. Add one self-contained notebook that exports the existing evaluation pairs as an EvalPort suite and ResultSet, validates them, and is listed in the README.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, mongodb, python
Domain
databases, machine-learning, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.