Dynamic Quality-Score-Based Soft Sampling

What:
Add a SoftSamplingStage that samples documents proportional to a quality score rather than applying a hard binary threshold. Higher-scored documents appear more often; lower-scored less often — but none are completely discarded.

Why:
SOTA models are using score-based soft sampling rather than hard filters. Hard thresholds discard potentially useful documents; soft sampling preserves diversity while biasing toward quality. Avoids the precision/recall tradeoff of hard cutoffs.

Definition of Done:

• SoftSamplingStage under nemo_curator/stages/text/
• Reads score from configurable metadata field
• Configurable sampling function: linear, sigmoid, power (default: linear)
• Configurable min/max sampling probability bounds
• Ray-native; no global shuffle required
• Emits before/after score distribution comparison in stats
• Test: verify output score distribution is statistically higher than input

Start by reading the existing stages and tests under nemo_curator/stages/text/ to identify the project’s Ray-native stage and stats patterns. Done means a SoftSamplingStage supports configurable score metadata, linear/sigmoid/power sampling and probability bounds, avoids a global shuffle, reports before/after distributions, and verifies statistically higher output scores.