NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer
Normalize null judge detail lists only for positive verdicts
@memadi-nv is already working on this.
Since Sep 15, 2026.
- Dominant language
- Python
- Stars
- 122
- Forks
- 17
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 11
Description
Problem
LLM judge responses may emit an explicit JSON null for a detail list whose documented empty state is an empty array. Pydantic defaults apply when a field is omitted, but not when it is present as null, so an otherwise usable positive judgment becomes unavailable.
This is related to the model-output drift handled in #268 and the conservative rewrite-evaluator normalization in #269.
Scope
Apply verdict-aware normalization to these evaluation schemas:
- DetectionJudgmentSchema.invalid_entities
- TypeFidelityJudgmentSchema.invalid_replacements
- RelationalConsistencyJudgmentSchema.relations
- AttributeFidelityJudgmentSchema.entities
Required behavior
Normalize null to an empty list only when the associated verdict is positive:
- all_valid=true plus invalid_entities=null becomes invalid_entities=[]
- all_valid=true plus invalid_replacements=null becomes invalid_replacements=[]
- all_consistent=true plus relations=null becomes relations=[]
- all_valid=true plus entities=null becomes entities=[]
Keep negative or inconsistent payloads invalid:
- a negative verdict plus a null or empty detail list must not be accepted as a complete judgment
- malformed output must continue to surface as judge unavailable rather than fabricating a verdict
Do not normalize nested RelationCheck.entities or AttributeCheck.attributes_checked; an emitted check requires those details.
Acceptance criteria
- Add model-level validation implementing the positive-verdict-only rule for all four schemas.
- Add parameterized tests covering positive/null, positive/empty, negative/null, negative/empty, and valid negative/non-empty cases.
- Preserve the current postprocessing behavior for malformed judgments.
- No public API changes.
- make format-check and make typecheck pass.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.