NVIDIA-NeMo / NVIDIA-NeMo/Curator

Optional EvalPort interop for QualityClassifier scores

Open
#2,329 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community-request waiting-on-maintainers
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Hi NeMo Curator team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Opening this as an issue per CONTRIBUTING.md, since it's a small, concrete proposal rather than a larger design change.

I installed nemo-curator and read the real classifier stage rather than guessing (nemo_curator/stages/text/classifiers/quality.py):

class QualityClassifier(DistributedDataClassifier):
    def __init__(
        self,
        label_field: str = "quality_pred",
        score_field: str | None = None,
        text_field: str = "text",
        filter_by: list[str] | None = None,
        ...
    ): ...

Running this stage over a dataset writes a label_field (default "quality_pred") and, when set, a score_field (the model's probability) onto every row, keyed off text_field. That's a direct match for EvalPort's other half: each row of text is a TestCase, and the quality_pred/probability pair QualityClassifier attaches to it is exactly the shape of an EvalPort Grader result — a per-TestCase label plus a numeric score — so a curated dataset's classifier output could become a ResultSet without inventing field names.

Two ways I could see this landing, and I don't have a strong preference:

  1. A standalone nemo-curator-openeval-adapter package in the EvalPort repo, depending on nemo-curator as a normal dependency. Zero footprint on this repo.
  2. A small optional module inside this repo if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nemo_curator/stages/text/classifiers/quality.py and CONTRIBUTING.md to understand the classifier interface and contribution constraints. Resolve whether the integration belongs in a standalone EvalPort adapter or an optional Curator module, then define tests that validate the resulting output against EvalPort's actual JSON Schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.