NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner

Jinja2 templates cannot reference columns created by PRE_BATCH processors

Open
#394 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.2k
Forks
211
Avg merge
2d 6h
Merged PRs (30d)
40

Description

Bug Description

Jinja2 {{ }} references in LLM column prompts fail when the referenced column is created by a PRE_BATCH processor. The compiler validates templates against the raw seed schema, which doesn't include columns added at runtime by processors.

Steps to Reproduce

  1. Create a workflow with a seed dataset that has columns [a, b]
  2. Add a PRE_BATCH processor that creates a new column c
  3. Define a downstream LLM column whose prompt references {{ c }}
  4. Run the workflow

Expected Behavior

The compiler should recognize that column c will exist after the PRE_BATCH processor runs, and allow {{ c }} in downstream prompts.

Actual Behavior

The compiler rejects the template because c doesn't exist in the raw seed data. The validation happens at compile time against the raw schema, before any processors run.

Root Cause

The compiler discovers seed columns from the raw seed reader and validates Jinja2 templates against that raw schema. PRE_BATCH processors can add columns at runtime, but the compiler has no way to know about them.

DropColumnsProcessor already implicitly declares removed columns via its config - the builder uses it to mark columns with drop=True at build time. But there's no equivalent mechanism for declaring added columns.

Proposed Fix

PRE_BATCH processors should declare which columns they add/remove so the compiler can compute the post-processor column set:

class ProcessorConfig(ConfigBase):
    processor_type: str
    columns_added: list[str] = []
    columns_removed: list[str] = []

The compiler would adjust the column set after seed column discovery: remove declared drops, add SeedDatasetColumnConfig entries for declared additions. Template validation and DAG resolution would then see the final schema.

This only applies to PRE_BATCH processors - POST_BATCH and AFTER_GENERATION processors don't need this since no downstream generators depend on their output schema.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the compiler's raw seed reader and the ProcessorConfig/DropColumnsProcessor handling described in the issue. Trace how the compiler builds the column set and validates Jinja2 templates and DAG dependencies. Done means PRE_BATCH additions and removals are reflected in the post-processor schema, allowing downstream references such as {{ c }} while leaving POST_BATCH and AFTER_GENERATION behavior unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.