NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer

feat: quick experimentation harness for testing configs and prompts

Open
#111 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement medium priority
Dominant language
Python
Stars
122
Forks
17
Avg merge
4d 2h
Merged PRs (30d)
11

Description

Summary

There is no lightweight way to quickly test a config change, prompt edit, or model swap against a small dataset without running the full pipeline. This makes iteration slow and expensive.

Current state

The existing test suite (tests/engine/) contains 22 unit tests that mock LLM calls. There is no tool for running live end-to-end experiments with real models against a small slice of data and comparing results across runs.

Proposed harness

A CLI command or notebook (or both) that supports:

  • Run a named config preset or inline config against N records from a dataset
  • Diff output columns (leakage_mass, utility_score, judge, needs_human_review) across two runs side by side
  • Print a summary table: avg leakage, avg utility, avg judge, # needing repair, # needing human review, # errors
  • Optionally pin a model or override defaults without editing source files
  • Works with any of the existing test datasets (TAB, BIO, RAT-Bench, Enron)

Why this matters

Right now testing a prompt change requires: editing code → reinstalling → running a full notebook → manually inspecting output. A harness would let Amy and others iterate on prompts and configs directly without needing to touch the library internals.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing tests in tests/engine/ and the available TAB, BIO, RAT-Bench, and Enron datasets. Determine whether a CLI command, notebook, or both best fits the proposed workflow, then define how configs, model overrides, run comparisons, and summary statistics will be exposed. Done means a small dataset can be run against a config and results can be compared with the listed output columns and metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.