[Feature Request]: Add conditional applicability per validation rule in Data Validator
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Description
Today, the Data Validator step applies each configured validation rule to every incoming row.
In some cases, this creates unnecessary complexity in the pipeline. A common scenario is when most validation rules must be applied to all rows, but one or more specific rules should only be applied when a condition is met.
For example:
Validation A: apply to all rows
Validation B: apply to all rows
Validation C: apply only when a given field or business condition matches
At the moment, the workaround is usually to split the flow with Filter Rows and use additional Data Validator steps. This works, but it makes the pipeline more verbose, harder to maintain, and can duplicate validation logic.
Proposal
Add an optional condition for each validation rule inside the Data Validator step.
Suggested behavior:
If no condition is defined, the validation rule applies to all rows
If a condition is defined, that validation rule is evaluated only when the condition is true
If the condition is false, that rule is skipped for the current row
This is not a request to add general row filtering inside Data Validator.
The goal is specifically to support conditional validation per rule.
Example use cases
Validate CPF only when person_type = 'F'
Validate CNPJ only when person_type = 'J'
Require cancellation date only when status = 'CANCELLED'
Apply state-specific validation only when state = 'SP'
Benefits
Reduces the need for extra Filter Rows steps
Reduces the need for duplicated Data Validator steps
Keeps related validation rules together in one place
Improves pipeline readability and maintainability
Better supports business rules where some validations are only applicable in specific contexts
Possible UI approach
Add an optional field per validation rule, such as:
Apply when
Condition
Validation condition
If empty, the rule applies to all rows.
Notes
A key design point is to keep the feature scoped to rule applicability, not general flow control.
This would preserve the current responsibility of the Data Validator step while making it more flexible for conditional business validations.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start by locating the Data Validator transform and its validation-rule configuration and UI entry points. Review how rules are evaluated for each incoming row and how conditions are represented elsewhere, then define and test the per-rule behavior: no condition applies universally, a true condition evaluates the rule, and a false condition skips it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100