NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer

Normalize null judge detail lists only for positive verdicts

Open
#270 0 comments 0 reactions 1 assignee View on GitHub

@memadi-nv is already working on this.

Since Sep 15, 2026.

Dominant language
Python
Stars
122
Forks
17
Avg merge
4d 2h
Merged PRs (30d)
11

Description

Problem

LLM judge responses may emit an explicit JSON null for a detail list whose documented empty state is an empty array. Pydantic defaults apply when a field is omitted, but not when it is present as null, so an otherwise usable positive judgment becomes unavailable.

This is related to the model-output drift handled in #268 and the conservative rewrite-evaluator normalization in #269.

Scope

Apply verdict-aware normalization to these evaluation schemas:

  • DetectionJudgmentSchema.invalid_entities
  • TypeFidelityJudgmentSchema.invalid_replacements
  • RelationalConsistencyJudgmentSchema.relations
  • AttributeFidelityJudgmentSchema.entities

Required behavior

Normalize null to an empty list only when the associated verdict is positive:

  • all_valid=true plus invalid_entities=null becomes invalid_entities=[]
  • all_valid=true plus invalid_replacements=null becomes invalid_replacements=[]
  • all_consistent=true plus relations=null becomes relations=[]
  • all_valid=true plus entities=null becomes entities=[]

Keep negative or inconsistent payloads invalid:

  • a negative verdict plus a null or empty detail list must not be accepted as a complete judgment
  • malformed output must continue to surface as judge unavailable rather than fabricating a verdict

Do not normalize nested RelationCheck.entities or AttributeCheck.attributes_checked; an emitted check requires those details.

Acceptance criteria

  • Add model-level validation implementing the positive-verdict-only rule for all four schemas.
  • Add parameterized tests covering positive/null, positive/empty, negative/null, negative/empty, and valid negative/non-empty cases.
  • Preserve the current postprocessing behavior for malformed judgments.
  • No public API changes.
  • make format-check and make typecheck pass.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.