NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner
Allow user to keep record fields that failed to parse as `null`.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 211
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 40
Description
Priority Level
Medium (Nice to have)
Is your feature request related to a problem? Please describe.
Currently, failed parses from e.g. LLMCodeColumnConfig give an error like
[WARNING] ⚠️ Generation for record at index N failed. Will omit this record from the dataset.
|----------
| Cause: The provided output schema was unable to be parsed from model 'model' responses while running generation for column 'column'.
| Solution: This is most likely temporary as we make additional attempts. If you continue to see more of this, simplify or modify the output schema for structured output and try again. If you are attempting token-intensive tasks like generations with high-reasoning effort, ensure that max_tokens in the model config is high enough to reach completion.
|----------
The trace (if set up to be exported) is also lost.
Describe the solution you'd like
Ideally, users should have the option to keep these records in the dataset with the json parsed to null. Three reasons:
- Some use-cases (with the ORDERED strategy) expect one output for every seed - this violates that.
- The
dropstrategy means losing other work done within the record, which might be useful to the user. - The trace is very useful in debugging parse failures.
Describe alternatives you've considered
Writing my own parser which also permits arbitrary text and parses to null - quite complicated though.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how LLMCodeColumnConfig parse failures are handled and where failed records are omitted from dataset generation. Define how an opt-in setting should retain those records with a null parsed value while preserving their traces, then verify the behavior for ORDERED and drop strategies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100