Line-Level Quality Filtering

What:
Add a stage that filters or removes individual low-quality lines within documents — rather than discarding entire documents — then enforces a word-removal ratio threshold (discard doc if too many lines were stripped). Targets boilerplate lines, nav menus, cookie notices, tracking scripts, and spam lines that survive document-level filters.

Why:
FLUX (March 2026) introduces a 4-stage pipeline where line-level cleaning is the critical step that enables simultaneously achieving higher quality and higher token retention vs. prior approaches. Surgical line removal outperforms whole-document rejection because most boilerplate-heavy documents still contain valuable content.

Definition of Done:

• LineLevelQualityFilter stage under nemo_curator/stages/text/filters/
• Configurable per-line removal rules: min line length, max repetition ratio, URL-only lines, nav/menu pattern regex, boilerplate string matching
• Word removal ratio enforcement: if removed_tokens / total_tokens > threshold (default: 0.3), discard document
• Reports per-rule removal statistics across dataset
• Composable with existing document-level filters (runs after, not instead of)
• Ray-native; operates line-by-line on partitioned datasets without full materialization
• Performance test
• End-to-end test: verify known boilerplate-heavy documents are cleaned, not discarded

Start by reading the existing stages under nemo_curator/stages/text/filters/ and determine how Ray-native, partitioned processing is structured. Implement the LineLevelQualityFilter requirements, including configurable removal rules, the 0.3 word-removal threshold, and per-rule statistics. Done means performance and end-to-end tests show boilerplate lines are removed while valuable documents are retained.