NVIDIA / NVIDIA/NeMo-Retriever

harness: add first-class dataset and ground-truth paths for portable single runs

Open
#2,390 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Problem

retriever harness run <benchmark> correctly starts from code-owned benchmark defaults, and --set is useful for expert tuning such as changing query.top_k. However, a developer running one registered benchmark on a workstation without the canonical dataset mount must currently know internal configuration paths:

uv run --frozen --project nemo_retriever retriever harness run jp20_beir \
  --set dataset.path=/raid/data/jp20 \
  --set evaluation.dataset_name=/raid/data/jp20_query_gt.csv

Machine-local corpus and evaluation inputs are normal operator inputs, not experimental benchmark tuning. Operators also should not need to know whether a local queries/qrels source is represented by dataset.query_file or evaluation.dataset_name.

This surfaced during second-host validation of the portable nightly workflow: the nightly dataset map handles host paths, but the direct single-benchmark CLI still exposes internal override keys.

Proposed interface

retriever harness run jp20_beir \
  --dataset-path /raid/data/jp20 \
  --ground-truth /raid/data/jp20_query_gt.csv

For an evaluation-free smoke run:

retriever harness run jp20_smoke \
  --dataset-path /raid/data/jp20

Design notes

  • Keep both options optional so existing registry defaults remain unchanged.
  • Keep --set as the advanced escape hatch for benchmark tuning.
  • Model these as typed run inputs at the benchmark-resolution seam rather than constructing internal --set strings in the Typer command.
  • --ground-truth should supply the local queries/qrels input and hide the current dataset/evaluation representation details.
  • Define unambiguous precedence. Preferably fail early when a first-class flag conflicts with the corresponding --set key.
  • Validate local paths before model, Ray, or GPU initialization.
  • Record the final resolved paths in resolved_benchmark.json.
  • Remote evaluation datasets such as ViDoRe should remain unchanged.

Acceptance criteria

  • retriever harness run jp20_beir continues to use the registered defaults.
  • --dataset-path PATH overrides the registered corpus location.
  • --ground-truth PATH provides the local queries/qrels source without requiring internal override keys.
  • jp20_smoke --dataset-path PATH works without a ground-truth argument.
  • Missing or unreadable inputs fail during resolution with actionable messages.
  • Conflicting first-class and generic overrides fail clearly.
  • resolved_benchmark.json contains the resolved corpus and ground-truth paths.
  • Existing --set, runfile, runset, nightly, and remote ViDoRe behavior remains compatible.
  • User documentation includes a portable single-benchmark example.

Scope

This makes registered benchmarks portable across machines with different local paths. It does not attempt to infer arbitrary corpus formats: the selected benchmark still defines input type, loader, ingest/query configuration, and evaluation semantics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the harness run benchmark-resolution seam and trace how Typer options and --set overrides become resolved benchmark inputs. Check the existing run, runset, nightly, and remote ViDoRe paths before adding the portable inputs. Done means validated paths, clear conflict handling, recorded resolved paths, preserved compatibility, and updated user documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.