DataQuality: rowsWith* helpers fail plan generation on class mapped queries
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 112
- Forks
- 261
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 96
Description
Bug Report
The meta::external::dataquality::rowsWith* helpers fail plan generation when the
DataQualityRelationValidation query is a class mapped query — Class.all()->project(...)->from(mapping, runtime).
They work when the query is a relation query — #>{db.table}#->select(...)->from(runtime).
This affects all six helpers: rowsWithEmptyColumn, rowsWithNonEmptyColumn, rowsWithValueOutsideRange,
rowsWithNegativeValue, rowsWithColumnLongerThan, rowsWithColumnDiffersFromPattern.
These are exactly the assertions the DataQuality rule suggestion engine emits
(core_dataquality/generation/rule_suggestions.pure), and the class mapped form is the shape used by the
DataQuality test models, so this is likely to be hit by anyone adopting suggested rules on a mapped model.
Steps to Reproduce:
-
Take a
DataQualityRelationValidationwhose query is a class projection ending infrom(mapping, runtime)—
for exampledemo::simplePersonValidationin
legend-engine-xts-dataquality/legend-engine-xt-dataquality-api/src/test/resources/inputs/test-data.pure:query: |demo::Person.all()->project(~[id: x|$x.id, fullName: x|$x.fullName, emailAddress: x|$x.emailAddress]) ->from(demo::PersonMap, demo::PersonRuntime); -
Add a validation using any
rowsWith*helper:{ name: 'emailPresent'; description: 'Validates that every person has an email address'; assertion: rel|$rel->rowsWithEmptyColumn(~emailAddress)->assertRelationEmpty(~[id, fullName]); } -
POST pure/v1/dataquality/generatePlan(orexecute) for that validation.
Expected Result:
A plan is generated, as it is for the same assertion over a relation query, and executing it returns the rows
whose column is empty.
Actual Result:
HTTP 500 at plan generation:
PureExecutionException: Execution error at
(resource:/core/pure/router/routing/router_routing.pure line:582 column:124),
"Cannot cast a collection of size 0 to multiplicity [1]"
at CompiledSupport.toOneWithMessage(CompiledSupport.java:340)
at CompiledSupport.toOne(CompiledSupport.java:327)
at core_pure_router_routing_router_routing$25$1.value(core_pure_router_routing_router_routing.java:1758)
...
router_routing.pure:582 is inside the shouldBeRouted branch that maps a FunctionDefinition's parameters
onto routed values:
f:FunctionDefinition<Any>[1]| if($state.shouldBeRouted,
|$func->functionType().parameters->evaluateAndDeactivate()->map(
p | let class = $p.genericType.rawType;
if (!$class->toOne()->instanceOf(DataType) && ... )
$class is empty for one of the parameters, so $class->toOne() fails. shouldBeRouted is only true for the
class mapped path, which is why the relation query shape is unaffected.
This is not caused by #5091
Worth stating explicitly, since the symptom resembles the regression fixed in #5104:
- It still reproduces with
meta::pure::functions::relation::eval_ColSpec_1__T_1__Z_$0_1$_removed from
meta::pure::router::routing::shouldStopFunctions, i.e. with #5091's routing change undone. rowsWithValueOutsideRangefails identically, and it is declared<T>only — it has no inferred column type
parameter and was untouched by the type parameter fix in #5104.- A plain
filtervalidation on the same model and runtime generates and executes fine (HTTP 200), so the
model, mapping and runtime are sound.
Measured on the same build, class mapped model:
| validation | assertion | result |
|---|---|---|
idNotNegative |
filter(row | $row.id > 0) |
200 |
emailPresent |
rowsWithEmptyColumn(~emailAddress) |
500 |
ageInRange |
rowsWithValueOutsideRange(~age, 0, 200) |
500 |
personHasSampleRow |
filter -> join(LEFT) -> rename -> rowsWithEmptyColumn |
500 |
Environment:
master at 4f1b3eba7ab (and with #5104 applied — #5104 fixes the relation query shape only), legend-pure
5.96.0, JDK 17, H2.
Additional Context:
The likely reason this went unnoticed: no test in the repository takes a DataQualityRelationValidation as far
as an ExecutionPlan on the class mapped path. doRelationTest stops at lambda generation, the helper tests in
dataquality_relation_helper_test.pure run over an in-memory #TDS, and
DataQualityValidationArtifactGenerationExtension.generate swallows every exception. #5104 adds plan generation
and execution coverage, but over a relation query, so it does not cover this path.
A useful first diagnostic would be to print which function and which parameter has the empty rawType at
router_routing.pure:582 — the same change made to resolveResolvedTypeParameters in #5104, applied here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at meta::pure::router::routing::router_routing.pure:582 and diagnose which routed function parameter has an empty rawType, following the diagnostic approach used in resolveResolvedTypeParameters for #5104. Reproduce with demo::simplePersonValidation in legend-engine-xts-dataquality/legend-engine-xt-dataquality-api/src/test/resources/inputs/test-data.pure, then cover class-mapped plan generation and execution for the rowsWith* helpers. Done means the class-mapped validation generates a plan and executes successfully, while relation queries remain covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100