NVIDIA-NeMo / NVIDIA-NeMo/DataDesignerPlugins

Extend ddp new to scaffold all DataDesigner plugin types

Open
#22 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement plugin tap
Dominant language
Python
Stars
7
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Parent epic: #15

Depends on: #21

Why

The runtime plugin system supports column generators, seed readers, and processors. The current scaffold emits only a column-generator package, so DDPlugins is not yet a complete reference tap authoring workflow.

CLI contract

Extend ddp new to:

uv run ddp new my-plugin --type column-generator
uv run ddp new my-plugin --type seed-reader
uv run ddp new my-plugin --type processor

Backwards compatibility:

uv run ddp new my-plugin

must continue to create a column-generator plugin and the help text must say that this is the default.

Generated column-generator template

  • Config class subclasses data_designer.config.base.SingleColumnConfig.
  • Discriminator field is column_type: Literal["<slug>"] = "<slug>".
  • Implementation class subclasses ColumnGeneratorFullColumn[ConfigClass].
  • Plugin object uses PluginType.COLUMN_GENERATOR.
  • Test imports plugin and calls assert_valid_plugin(plugin).

Generated seed-reader template

  • Config class subclasses data_designer.config.seed_source.FileSystemSeedSource.
  • Discriminator field is seed_type: Literal["<slug>"] = "<slug>".
  • Implementation class subclasses data_designer.engine.resources.seed_reader.FileSystemSeedReader[ConfigClass].
  • Generated implementation should be a minimal placeholder that imports cleanly and clearly marks get_dataset_uri / DuckDB connection behavior as TODOs rather than pretending to be production-ready.
  • Plugin object uses PluginType.SEED_READER.
  • Test imports plugin and calls assert_valid_plugin(plugin).

Generated processor template

  • Config class subclasses data_designer.config.base.ProcessorConfig.
  • Discriminator field is processor_type: Literal["<slug>"] = "<slug>".
  • Implementation class subclasses data_designer.engine.processing.processors.base.Processor[ConfigClass].
  • Generated implementation overrides process_after_batch and returns data unchanged with a TODO.
  • Plugin object uses PluginType.PROCESSOR.
  • Test imports plugin and calls assert_valid_plugin(plugin) if DataDesigner validates processor plugins through that helper; otherwise the test should cover public import and plugin object construction while documenting the DataDesigner validation gap.

Shared generated behavior

  • Use tap config from #21 for package prefix, author, repository URL, docs URL examples, and data-designer dependency.
  • Use absolute imports only.
  • Use Python 3.10+ annotations.
  • Generate README and docs text that says “plugin type” or “entry point,” not “column type,” unless the generated plugin is specifically a column generator.
  • Add scaffold golden tests for file tree and important file contents for all three plugin types.
  • Add CLI tests for valid types, invalid type errors, and default behavior.

Acceptance criteria

  • ddp new my-plugin --type column-generator creates a valid column-generator plugin.
  • ddp new my-plugin --type seed-reader creates a valid seed-reader plugin.
  • ddp new my-plugin --type processor creates a valid processor plugin or a documented processor-validation-limited scaffold if DataDesigner lacks full validation.
  • uv run pytest devtools/ddp/tests/test_cli.py devtools/ddp/tests/test_scaffold.py -q covers all variants.
  • Generated docs and plugin cards do not label seed readers or processors as column types.

Dependencies

  • Depends on: #21.
  • Related DataDesigner dependency marker: DD-CORE-PROCESSOR-PLUGIN-VALIDATION.
  • Blocks: external tap authoring workflow docs in #25.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dependency #21 and the ddp new entry point, then inspect devtools/ddp/tests/test_cli.py and devtools/ddp/tests/test_scaffold.py. Cover column-generator, seed-reader, and processor scaffolds, including valid and invalid types and the default behavior. Done means the generated trees, contents, docs, and plugin tests are covered by uv run pytest devtools/ddp/tests/test_cli.py devtools/ddp/tests/test_scaffold.py -q.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.