apache / apache/hop

[Feature Request]: Data Validator - Enable Rules as a Static Schema Definition

Open
#7,571 4 comments 0 reactions 0 assignees View on GitHub
awaiting triage Hop Gui P2
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
19h
Merged PRs (30d)
227

Description

### What would you like to happen?

## Summary

Enhance the **Data Validator** transform to support loading validation rules from a **Static Schema Definition** (Metadata Perspective) using a JSON document.

Instead of manually configuring every validation rule inside the transform, users would be able to reference a JSON schema stored in the Metadata Perspective. The transform would interpret the schema and create the validation rules at runtime.

This would allow validation logic to be externalised from the pipeline while remaining version-controlled, reusable and understandable by non-Hop users.

---

## Problem

The current Data Validator requires all validation rules to be manually configured within the transform.

This becomes difficult when:

- Validation rules are generated externally.
- Multiple test suites require different validation sets.
- The same validation logic is reused across many pipelines.
- Governance teams need to review or approve validation rules.
- Validation rules evolve independently of pipeline logic.

Changing validation rules currently requires editing the pipeline itself.

---

## Proposed Solution

Add support for selecting a **Static Schema Definition** from the Metadata Perspective.

The selected schema would contain the validator definitions in JSON.

The Data Validator would:

1. Load the JSON schema.
2. Convert it internally into the existing validator definitions.
3. Execute validations exactly as it does today.

This preserves full backwards compatibility while allowing validations to be managed independently of the pipeline.

---

## Example

### Current XML representation

```xml


AAA
ABC

ddmm
String
Y
.
@#$
df
Error1
Error 1
,
Y
5
234
234
234
String1
Y
Y
Y
.*
.*
field
src
34
QW
Sample1


df
df

234rt
ert
Y
234
2wer
wer
Error2
Error2
34rt
Y
234r
we
wer
wer
String2
Y
Y
Y
we
we
dfv
we
er
Sample2

```

### Equivalent JSON

```json
[
{
"allowedValue": [
"AAA",
"ABC"
],
"conversionMask": "ddmm",
"dataType": "String",
"dataTypeVerified": true,
"decimalSymbol": ".",
"endString": "@#$",
"endStringNotAllowed": "df",
"errorCode": "Error1",
"errorDescription": "Error 1",
"groupingSymbol": ",",
"isSourcingValues": true,
"maxLength": 5,
"maxValue": "234",
"minLength": 234,
"minValue": "234",
"name": "String1",
"nullAllowed": true,
"onlyNullAllowed": true,
"onlyNumericAllowed": true,
"regularExpression": ".*",
"regularExpressionNotAllowed": ".*",
"sourcingField": "field",
"sourcingTransform": "src",
"startString": "34",
"startStringNotAllowed": "QW",
"validationName": "Sample1"
},
{
"allowedValue": [
"df",
"df"
],
"conversionMask": "234rt",
"dataType": "ert",
"dataTypeVerified": true,
"decimalSymbol": "234",
"endString": "2wer",
"endStringNotAllowed": "wer",
"errorCode": "Error2",
"errorDescription": "Error2",
"groupingSymbol": "34rt",
"isSourcingValues": true,
"maxLength": "234r",
"maxValue": "we",
"minLength": "wer",
"minValue": "wer",
"name": "String2",
"nullAllowed": true,
"onlyNullAllowed": true,
"onlyNumericAllowed": true,
"regularExpression": "we",
"regularExpressionNotAllowed": "we",
"sourcingField": "dfv",
"startString": "we",
"startStringNotAllowed": "er",
"validationName": "Sample2"
}
]
```

---

## Benefits

### Reusable validation

The same validation schema can be used across many pipelines without duplication.

### Easier testing

Entire validation suites can be swapped simply by selecting a different Static Schema Definition.

For example:

- Development validations
- Integration validations
- Production validations
- Client-specific validations

No pipeline changes required.

### Reusable validation pipelines

A generic validation pipeline could be built where only three metadata objects change:

- Input schema
- Validation schema
- Output schema

This enables highly reusable, metadata-driven pipelines.

### Better governance

JSON is significantly easier for business users, governance teams and data stewards to review than transform XML.

Validation rules can be:

- Peer reviewed
- Version controlled
- Automatically generated
- Validated using JSON Schema
- Managed outside Hop

### Easier upgrades

Validation logic can evolve independently from pipeline logic.

Changing validation rules would not require opening, modifying or redeploying pipelines.

### Better interoperability

External applications could generate validation definitions programmatically, allowing Hop to consume validation rules from governance platforms, metadata repositories or code generators.

This also opens the possibility of generating validation schemas from other metadata sources or enterprise data catalogues.

---

## Backwards Compatibility

This feature could be completely optional.

Existing Data Validator transforms would continue to work exactly as they do today.

Users could choose between:

- Embedded validator definitions (current behaviour)
- Static Schema Definition (JSON)

Image

Image

Image

### Issue Priority

Priority: 2

### Issue Component

Component: Hop Gui

Contributor guide

Open the contributing guide

Research direction

Start by locating the Data Validator transform and the Static Schema Definition support in the Metadata Perspective. Review how existing validator definitions are configured and how metadata objects are selected; done means JSON rules can be loaded and used while existing embedded definitions continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.