NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner

Allow user to keep record fields that failed to parse as `null`.

Open
#362 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.2k
Forks
211
Avg merge
2d 6h
Merged PRs (30d)
40

Description

Priority Level

Medium (Nice to have)

Is your feature request related to a problem? Please describe.

Currently, failed parses from e.g. LLMCodeColumnConfig give an error like

[WARNING] ⚠️ Generation for record at index N failed. Will omit this record from the dataset.
  |----------
  | Cause: The provided output schema was unable to be parsed from model 'model' responses while running generation for column 'column'.
  | Solution: This is most likely temporary as we make additional attempts. If you continue to see more of this, simplify or modify the output schema for structured output and try again. If you are attempting token-intensive tasks like generations with high-reasoning effort, ensure that max_tokens in the model config is high enough to reach completion.
  |----------

The trace (if set up to be exported) is also lost.

Describe the solution you'd like

Ideally, users should have the option to keep these records in the dataset with the json parsed to null. Three reasons:

  1. Some use-cases (with the ORDERED strategy) expect one output for every seed - this violates that.
  2. The drop strategy means losing other work done within the record, which might be useful to the user.
  3. The trace is very useful in debugging parse failures.
Describe alternatives you've considered

Writing my own parser which also permits arbitrary text and parses to null - quite complicated though.

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how LLMCodeColumnConfig parse failures are handled and where failed records are omitted from dataset generation. Define how an opt-in setting should retain those records with a null parsed value while preserving their traces, then verify the behavior for ORDERED and drop strategies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.