NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner
Add more stages to processors, and allow for custom processors via callbacks
@andreatnvidia is already working on this.
Since Dec 22, 2025.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 211
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 40
Description
Is your feature request related to a problem? Please describe.
Right now processors only run after each batch is generated. It should also be possible for the execution to happen pre/post-generation.
Also, users should be able to implement custom processors easily via callbacks (same as e.g., the LocalCallableValidator).
Describe the solution you'd like
There are a couple of ways of going about this. Following the original plan, we would allow the user to pick different stages for any given processor.
However, it might be more interesting to have processors implementing all different stages via different methods - instead of process, preprocess/postprocess/process_after_batch/etc.
That because a processor could actually require different procedures to happen at different times (e.g., save artifacts after batch is generated, and collect all of them at the end).
Describe alternatives you've considered
n/a
Additional context
n/a
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.