NVIDIA / NVIDIA/NeMo-Retriever
harness: add first-class dataset and ground-truth paths for portable single runs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Problem
retriever harness run <benchmark> correctly starts from code-owned benchmark defaults, and --set is useful for expert tuning such as changing query.top_k. However, a developer running one registered benchmark on a workstation without the canonical dataset mount must currently know internal configuration paths:
uv run --frozen --project nemo_retriever retriever harness run jp20_beir \
--set dataset.path=/raid/data/jp20 \
--set evaluation.dataset_name=/raid/data/jp20_query_gt.csv
Machine-local corpus and evaluation inputs are normal operator inputs, not experimental benchmark tuning. Operators also should not need to know whether a local queries/qrels source is represented by dataset.query_file or evaluation.dataset_name.
This surfaced during second-host validation of the portable nightly workflow: the nightly dataset map handles host paths, but the direct single-benchmark CLI still exposes internal override keys.
Proposed interface
retriever harness run jp20_beir \
--dataset-path /raid/data/jp20 \
--ground-truth /raid/data/jp20_query_gt.csv
For an evaluation-free smoke run:
retriever harness run jp20_smoke \
--dataset-path /raid/data/jp20
Design notes
- Keep both options optional so existing registry defaults remain unchanged.
- Keep
--setas the advanced escape hatch for benchmark tuning. - Model these as typed run inputs at the benchmark-resolution seam rather than constructing internal
--setstrings in the Typer command. --ground-truthshould supply the local queries/qrels input and hide the current dataset/evaluation representation details.- Define unambiguous precedence. Preferably fail early when a first-class flag conflicts with the corresponding
--setkey. - Validate local paths before model, Ray, or GPU initialization.
- Record the final resolved paths in
resolved_benchmark.json. - Remote evaluation datasets such as ViDoRe should remain unchanged.
Acceptance criteria
-
retriever harness run jp20_beircontinues to use the registered defaults. -
--dataset-path PATHoverrides the registered corpus location. -
--ground-truth PATHprovides the local queries/qrels source without requiring internal override keys. -
jp20_smoke --dataset-path PATHworks without a ground-truth argument. - Missing or unreadable inputs fail during resolution with actionable messages.
- Conflicting first-class and generic overrides fail clearly.
-
resolved_benchmark.jsoncontains the resolved corpus and ground-truth paths. - Existing
--set, runfile, runset, nightly, and remote ViDoRe behavior remains compatible. - User documentation includes a portable single-benchmark example.
Scope
This makes registered benchmarks portable across machines with different local paths. It does not attempt to infer arbitrary corpus formats: the selected benchmark still defines input type, loader, ingest/query configuration, and evaluation semantics.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the harness run benchmark-resolution seam and trace how Typer options and --set overrides become resolved benchmark inputs. Check the existing run, runset, nightly, and remote ViDoRe paths before adding the portable inputs. Done means validated paths, clear conflict handling, recorded resolved paths, preserved compatibility, and updated user documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100