NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner

Add more stages to processors, and allow for custom processors via callbacks

Open
#159 2 comments 0 reactions 1 assignee View on GitHub

@andreatnvidia is already working on this.

Since Dec 22, 2025.

enhancement
Dominant language
Python
Stars
2.2k
Forks
211
Avg merge
2d 6h
Merged PRs (30d)
40

Description

Is your feature request related to a problem? Please describe.

Right now processors only run after each batch is generated. It should also be possible for the execution to happen pre/post-generation.
Also, users should be able to implement custom processors easily via callbacks (same as e.g., the LocalCallableValidator).

Describe the solution you'd like

There are a couple of ways of going about this. Following the original plan, we would allow the user to pick different stages for any given processor.

However, it might be more interesting to have processors implementing all different stages via different methods - instead of process, preprocess/postprocess/process_after_batch/etc.
That because a processor could actually require different procedures to happen at different times (e.g., save artifacts after batch is generated, and collect all of them at the end).

Describe alternatives you've considered

n/a

Additional context

n/a

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.