Classifier-Calibration Audit (Data-Quality Illusion)

What:
Add a diagnostic stage that audits whether a trained quality classifier (e.g., FineWeb-Edu-style fastText/LLM classifier) is capturing genuine quality signal or distributional artifacts. Tests: (1) whether filtered data improves language modeling on the reference set, (2) whether classifier scores correlate with held-out human quality judgments, (3) whether the classifier implicitly re-filters the reference dataset itself.

Why:
arXiv:2510.00866 proves that Classifier-based Quality Filtering improves downstream task performance but does not improve language modeling on the high-quality reference set. The classifier exploits distributional artifacts, not genuine quality. Without this audit, practitioners trust classifiers that may be overfitting to distribution shift rather than quality — leading to compounding errors across pipelines.

Definition of Done:

• ClassifierCalibrationAudit under nemo_curator/audit/
• Test 1 — Reference LM check: trains a 125M proxy on filtered data vs. unfiltered, computes perplexity on reference corpus (not benchmark); flags if
  filtered model is worse
• Test 2 — Human correlation: accepts optional human-annotated quality sample; reports Spearman ρ between classifier score and human rating
• Test 3 — Reference contamination check: applies classifier to the reference dataset itself; reports what fraction the classifier would remove (should be
  near 0% for a calibrated classifier)
• Outputs an audit report JSON with pass/warn/fail per test
• Integration test: a deliberately miscalibrated classifier (trained on distribution-shifted data) should trigger warnings
• Tutorial: auditing a custom-trained classifier before deploying to production pipeline

Start by surveying the new nemo_curator/audit/ area and the classifier and reference-corpus entry points involved in the Definition of Done. Map the three requested checks, the pass/warn/fail JSON report, the deliberately miscalibrated integration case, and the custom-classifier tutorial before implementation; done means all three diagnostics and their integration coverage are present.