finos / finos/legend-engine

DataQuality: rowsWith* helpers fail plan generation on class mapped queries

Open
#5,106 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
112
Forks
261
Avg merge
1d 13h
Merged PRs (30d)
96

Description

Bug Report

The meta::external::dataquality::rowsWith* helpers fail plan generation when the
DataQualityRelationValidation query is a class mapped query — Class.all()->project(...)->from(mapping, runtime).
They work when the query is a relation query — #>{db.table}#->select(...)->from(runtime).

This affects all six helpers: rowsWithEmptyColumn, rowsWithNonEmptyColumn, rowsWithValueOutsideRange,
rowsWithNegativeValue, rowsWithColumnLongerThan, rowsWithColumnDiffersFromPattern.

These are exactly the assertions the DataQuality rule suggestion engine emits
(core_dataquality/generation/rule_suggestions.pure), and the class mapped form is the shape used by the
DataQuality test models, so this is likely to be hit by anyone adopting suggested rules on a mapped model.

Steps to Reproduce:
  1. Take a DataQualityRelationValidation whose query is a class projection ending in from(mapping, runtime)
    for example demo::simplePersonValidation in
    legend-engine-xts-dataquality/legend-engine-xt-dataquality-api/src/test/resources/inputs/test-data.pure:

    query: |demo::Person.all()->project(~[id: x|$x.id, fullName: x|$x.fullName, emailAddress: x|$x.emailAddress])
             ->from(demo::PersonMap, demo::PersonRuntime);
    
  2. Add a validation using any rowsWith* helper:

    {
      name: 'emailPresent';
      description: 'Validates that every person has an email address';
      assertion: rel|$rel->rowsWithEmptyColumn(~emailAddress)->assertRelationEmpty(~[id, fullName]);
    }
    
  3. POST pure/v1/dataquality/generatePlan (or execute) for that validation.

Expected Result:

A plan is generated, as it is for the same assertion over a relation query, and executing it returns the rows
whose column is empty.

Actual Result:

HTTP 500 at plan generation:

PureExecutionException: Execution error at
  (resource:/core/pure/router/routing/router_routing.pure line:582 column:124),
  "Cannot cast a collection of size 0 to multiplicity [1]"
    at CompiledSupport.toOneWithMessage(CompiledSupport.java:340)
    at CompiledSupport.toOne(CompiledSupport.java:327)
    at core_pure_router_routing_router_routing$25$1.value(core_pure_router_routing_router_routing.java:1758)
    ...

router_routing.pure:582 is inside the shouldBeRouted branch that maps a FunctionDefinition's parameters
onto routed values:

f:FunctionDefinition<Any>[1]| if($state.shouldBeRouted,
    |$func->functionType().parameters->evaluateAndDeactivate()->map(
        p | let class = $p.genericType.rawType;
            if (!$class->toOne()->instanceOf(DataType) && ... )

$class is empty for one of the parameters, so $class->toOne() fails. shouldBeRouted is only true for the
class mapped path, which is why the relation query shape is unaffected.

This is not caused by #5091

Worth stating explicitly, since the symptom resembles the regression fixed in #5104:

  • It still reproduces with meta::pure::functions::relation::eval_ColSpec_1__T_1__Z_$0_1$_ removed from
    meta::pure::router::routing::shouldStopFunctions, i.e. with #5091's routing change undone.
  • rowsWithValueOutsideRange fails identically, and it is declared <T> only — it has no inferred column type
    parameter and was untouched by the type parameter fix in #5104.
  • A plain filter validation on the same model and runtime generates and executes fine (HTTP 200), so the
    model, mapping and runtime are sound.

Measured on the same build, class mapped model:

validation assertion result
idNotNegative filter(row | $row.id > 0) 200
emailPresent rowsWithEmptyColumn(~emailAddress) 500
ageInRange rowsWithValueOutsideRange(~age, 0, 200) 500
personHasSampleRow filter -> join(LEFT) -> rename -> rowsWithEmptyColumn 500
Environment:

master at 4f1b3eba7ab (and with #5104 applied — #5104 fixes the relation query shape only), legend-pure
5.96.0, JDK 17, H2.

Additional Context:

The likely reason this went unnoticed: no test in the repository takes a DataQualityRelationValidation as far
as an ExecutionPlan on the class mapped path. doRelationTest stops at lambda generation, the helper tests in
dataquality_relation_helper_test.pure run over an in-memory #TDS, and
DataQualityValidationArtifactGenerationExtension.generate swallows every exception. #5104 adds plan generation
and execution coverage, but over a relation query, so it does not cover this path.

A useful first diagnostic would be to print which function and which parameter has the empty rawType at
router_routing.pure:582 — the same change made to resolveResolvedTypeParameters in #5104, applied here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at meta::pure::router::routing::router_routing.pure:582 and diagnose which routed function parameter has an empty rawType, following the diagnostic approach used in resolveResolvedTypeParameters for #5104. Reproduce with demo::simplePersonValidation in legend-engine-xts-dataquality/legend-engine-xt-dataquality-api/src/test/resources/inputs/test-data.pure, then cover class-mapped plan generation and execution for the rowsWith* helpers. Done means the class-mapped validation generates a plan and executes successfully, while relation queries remain covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.