Portable eval interchange (EvalPort) as a dataset-loader / results third-party plugin

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request?

Low (would be nice)

Please provide a clear description of problem this feature solves

I want to run a NeMo Agent Toolkit nat eval suite against a test dataset that was authored for (or produced by) a different eval stack — e.g. a suite exported by Ragas, LangSmith, or a hand-written suite meant to be reusable across frameworks — without hand-converting it into NeMo Agent Toolkit's own question_key/answer_key JSON/JSONL/CSV/Parquet shape first, because I need to keep one canonical set of test cases that also runs unmodified against other eval tools.

I'm the maintainer of EvalPort (Apache-2.0), an open interchange format + Python/TS SDK (evalport-sdk) for portable LLM-eval test cases, graders, suites, and results. It already has 40 real adapter packages under adapters/ for tools NeMo Agent Toolkit already touches directly — for example ragas-openeval-adapter (Ragas is a registered evaluator dependency here) and weave-openeval-adapter (W&B Weave is credited in this repo's acknowledgements and has its own nvidia_nat_weave package). langsmith-openeval-adapter and phoenix-openeval-adapter also exist, both relevant given the native LangSmith integration and nvidia_nat_phoenix package here.

What EvalPort is not trying to do: it doesn't touch agent execution traces. I read through packages/nvidia_nat_atif/ATOF and see NeMo Agent Toolkit already has a real, mature answer for trajectory-level interchange (IntermediateStep → ATIF steps, with its own RFC lineage via harbor-framework/ATIF). EvalPort sits one layer up: the test-case/dataset definition, the grader/evaluator definition, suite composition, and the final scored-result shape — not the intermediate tool-call trace. So this proposal is scoped to complement ATIF, not duplicate it.

Concretely, today EvalDatasetBaseConfig (in packages/nvidia_nat_core/src/nat/data_models/dataset_handler.py) already supports pluggable dataset shapes via @register_dataset_loader(config_type=...) — json, jsonl, csv, parquet, xls, and custom are all registered the same way in packages/nvidia_nat_eval/src/nat/plugins/eval/dataset_loader/register.py, each just supplying a DatasetLoaderInfo(config=config, load_fn=..., description=...). That pattern is a natural fit for also reading (and, optionally, writing) EvalPort's format without touching the framework-agnostic core.

Describe your ideal solution

Given the new Third-Party Plugin Packages model this repo just introduced (Tavily/Redis/ATR as the current examples), I think the right shape for this is a provider-owned package following that exact convention rather than an in-tree change:

Surface	Value
GitHub repo	NeMo-Agent-Toolkit-evalport
Python distribution	nemo-agent-toolkit-evalport
Python import package	nat.plugins.evalport
Entry point	nat_evalport
Registered _type	evalport

A minimal sketch of the dataset-loader side, following the exact pattern in dataset_loader/register.py:

# nat/plugins/evalport/register.py
from evalport_sdk import load_suite  # returns list[dict] of EvalPort TestCase records

from nat.builder.builder import EvalBuilder
from nat.builder.dataset_loader import DatasetLoaderInfo
from nat.cli.register_workflow import register_dataset_loader
from nat.data_models.dataset_handler import EvalDatasetBaseConfig


class EvalDatasetEvalPortConfig(EvalDatasetBaseConfig, name="evalport"):
    """Load an EvalPort test suite as a NeMo Agent Toolkit eval dataset."""


def _read_evalport_suite(file_path, **kwargs):
    cases = load_suite(file_path)  # validates against the EvalPort suite schema
    # Map EvalPort's TestCase.input / .expected onto whatever
    # structure.question_key / structure.answer_key / id_key the
    # EvalDatasetStructureConfig on this config instance specifies,
    # same as the existing json/jsonl loaders do implicitly today.
    return pd.DataFrame([{
        "id": c["id"],
        "question": c["input"],
        "answer": c["expected"],
        **c.get("metadata", {}),
    } for c in cases])


@register_dataset_loader(config_type=EvalDatasetEvalPortConfig)
async def register_evalport_dataset_loader(config: EvalDatasetEvalPortConfig, builder: EvalBuilder):
    yield DatasetLoaderInfo(config=config, load_fn=_read_evalport_suite,
                             description="EvalPort interchange-format dataset loader")

Workflow config would then look like the existing csv/parquet examples in docs/source/improve-workflows/evaluate.md, just with _type: evalport.

On the output side, FileEvalCallback already writes a second, parallel file for ATIF (workflow_output_atif.json next to workflow_output.json) — the same pattern could optionally produce an EvalPort-format results file from EvalOutput/EvalOutputItem (id, score, reasoning, error map cleanly onto EvalPort's result schema), so a nat eval run becomes something other EvalPort adapters (Ragas, Weave, LangSmith, ...) can also read. I'd treat this as optional/secondary to the dataset-loader side, and I'm not assuming it's wanted.

I'm not asking for a PR to be merged — I'd like to know:

1. Whether the team would want this proposed through the third-party plugin Apply step described in that doc (name/scope/license/repo, then build against the template), given the naming table above.
2. Whether the dataset-loader direction alone is of interest, independent of the results-exporter half.
3. Anything about the real EvalDatasetStructureConfig mapping (question_key/answer_key/generated_answer_key/trajectory_key/expected_trajectory_key) I should account for that isn't obvious from reading the code.

Additional context

• EvalPort: https://github.com/adhabnr-ux/evalport (Apache-2.0, evalport-sdk for Python/TS, 40 merged adapters under adapters/)
• Comparable, already-merged adapters relevant to this repo's own dependencies: ragas-openeval-adapter, weave-openeval-adapter
• Reference points read in this repo: packages/nvidia_nat_core/src/nat/data_models/dataset_handler.py, packages/nvidia_nat_core/src/nat/data_models/evaluator.py, packages/nvidia_nat_eval/src/nat/plugins/eval/dataset_loader/register.py, packages/nvidia_nat_eval/src/nat/plugins/eval/data_models/evaluator_io.py, packages/nvidia_nat_eval/src/nat/plugins/eval/exporters/file_eval_callback.py, packages/nvidia_nat_atif/, docs/source/extend/third-party-plugins.md

— Sahi, independent contributor (not affiliated with NVIDIA)

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

Start with docs/source/extend/third-party-plugins.md and compare the existing registration pattern in packages/nvidia_nat_eval/src/nat/plugins/eval/dataset_loader/register.py with the dataset models in packages/nvidia_nat_core/src/nat/data_models/dataset_handler.py. Review the referenced evaluator and export files to establish whether the proposed EvalPort package fits the plugin API. Done requires a maintainer decision on the third-party dataset-loader scope and whether results export is wanted.