NVIDIA-NeMo / NVIDIA-NeMo/Curator
Optional EvalPort interop for QualityClassifier scores
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Hi NeMo Curator team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Opening this as an issue per CONTRIBUTING.md, since it's a small, concrete proposal rather than a larger design change.
I installed nemo-curator and read the real classifier stage rather than guessing (nemo_curator/stages/text/classifiers/quality.py):
class QualityClassifier(DistributedDataClassifier):
def __init__(
self,
label_field: str = "quality_pred",
score_field: str | None = None,
text_field: str = "text",
filter_by: list[str] | None = None,
...
): ...
Running this stage over a dataset writes a label_field (default "quality_pred") and, when set, a score_field (the model's probability) onto every row, keyed off text_field. That's a direct match for EvalPort's other half: each row of text is a TestCase, and the quality_pred/probability pair QualityClassifier attaches to it is exactly the shape of an EvalPort Grader result — a per-TestCase label plus a numeric score — so a curated dataset's classifier output could become a ResultSet without inventing field names.
Two ways I could see this landing, and I don't have a strong preference:
- A standalone
nemo-curator-openeval-adapterpackage in the EvalPort repo, depending onnemo-curatoras a normal dependency. Zero footprint on this repo. - A small optional module inside this repo if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with nemo_curator/stages/text/classifiers/quality.py and CONTRIBUTING.md to understand the classifier interface and contribution constraints. Resolve whether the integration belongs in a standalone EvalPort adapter or an optional Curator module, then define tests that validate the resulting output against EvalPort's actual JSON Schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100