Overhaul Iterate/Extract abstraction as concepts in favor of a simpler Stage that yields a Task
@sarahyurick is already working on this.
Since Feb 4, 2026.
Assessment
This issue has not been assessed yet.
Description
Description
PR #1458 (https://github.com/NVIDIA-NeMo/Curator/pull/1458) improves performance and stability by fusing document iterate + extract stages (benchmarked faster and less OOM-prone). During review, we identified a broader follow-up: the existing Iterate/Extract vocabulary and abstract classes are confusing and may overly constrain users to record-level extract operations.
Goal
Move toward a simpler, unified abstraction whose generic purpose is: “produce DocumentBatch from a list/group of files” (e.g., a ProcessingStage[FileGroupTask, DocumentBatch] style stage), enabling implementers to apply transformations in a flexible way (including vectorized operations where applicable), without forcing an Iterate/Extract split.
Why not in PR #1458
Iterate/Extract are used across current pipelines (cc/wiki/arxiv) and tutorials. A full refactor would touch many scripts, require updating tutorials, and needs careful testing—too large in scope for #1458.
Scope / Tasks
- Design a unified stage API for “file group -> document batch” production.
- Determine migration path from current Iterate/Extract:
- Identify impacted pipelines and tutorials.
- Provide an adapter/shim or staged deprecation plan.
- Refactor download+extract implementations to use the new unified approach.
- Update tutorials/docs and add/expand test coverage to ensure behavior remains correct.
- Decide whether/when to remove/deprecate old Iterate/Extract abstract classes.
Acceptance criteria
- A new unified stage exists and is usable for at least one end-to-end pipeline.
- Tutorials and core pipelines remain working (or are updated) with test coverage.
- Clear deprecation/migration guidance is documented for users of Iterate/Extract.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/Curator
-
community-request waiting-on-maintainers
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NVIDIA-NeMo/Curator#2375 ·
-
DocumentSplitter treats multi-character separators as regex (wrong splits, re.error crash on "++") Opencommunity-request waiting-on-maintainers
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/Curator#2374 ·
-
xenna_grafana_dashboard.json missing from the wheel — Xenna Grafana dashboard is never provisioned Opencommunity-request waiting-on-customer
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
NVIDIA-NeMo/Curator#2189 · 5 comments ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/Curator#1544 · 1 comment ·
-
community-request waiting-on-maintainers
Difficulty 2/5 1-3 hours Newbie friendliness 48/100
NVIDIA-NeMo/Curator#2397 ·
All issues in NVIDIA-NeMo/Curator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100