GoogleCloudPlatform / GoogleCloudPlatform/cloud-data-quality

Consistency Rules not working on Dataplex

Open
#254 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
283
Forks
59
PR merge metrics
No merged PRs in 30d

Description

Hello!

I'm using Data Quality Tasks on Dataplex and i'm using a rule of consistency (using a sql case when), and i get an error and looking for a example of rules of this dimension (consistency) i can't find an example of it.

This is my .yaml file:
---
rules:
CONSISTENCY_MINSS:
rule_type: CUSTOM_SQL_EXPR
dimension: consistency
params:
custom_sql_arguments:
- ref_data_dataset
- ref_data_table_id
custom_sql_expr: |-
case
when ($column IN (0,10,100)) or ($column is null) then true
else false end
from data a
inner join `$ref_data_project.$ref_data_dataset.$ref_data_table_id` b
on a.loc = b.loc
rule_dimensions:
- consistency
row_filters:
_PARTITIONTIME:
filter_sql_expr: |-
DATE(_PARTITIONTIME) = '2024-03-03'
rule_bindings:
MY-RULE-BINDING-CONSISTENCY-MINSS:
entity_uri: bigquery://projects/my-gcp-project-ENV/locations/US/datasets/my_gcp_dataset_ENV/tables/my_bigquery_tablename_ENV
column_id: column_name
row_filter_id: _PARTITIONTIME
rule_ids:
- CONSISTENCY_MINSS:
- ref_data_project: df-datalake-transformed-ENV
- ref_data_dataset: my_gcp_dataset_ENV
- ref_data_table_id: my_gcp_table_ENV
metadata:
manual_column_id: minss

**This is the error in the output on dataproc:**

metadata_registry_defaults:
dataplex:
projects:
locations:
lakes:
zones:

2024-03-12 18:35:10 gdpic-srvls-batch-9e830eba-2731-47cc-beeb-3fdf1ea8b20f-m clouddq[80] ERROR Failed to resolve Rule Binding ID 'MY-RULE-BINDING-CONSISTENCY-MINSS' with error:
Failed to resolve rule_id 'CONSISTENCY_MINSS' in rule_binding_id 'MY-RULE-BINDING-CONSISTENCY-MINSS' with error:
'list' object has no attribute 'get'
Traceback (most recent call last):
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/dq_rule_binding.py", line 213, in resolve_rule_sql_expr
rule.resolve_sql_expr()
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/dq_rule.py", line 122, in resolve_sql_expr
self.rule_sql_expr = self.rule_type.to_sql(self.params).safe_substitute()
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/rule_type.py", line 187, in to_sql
return to_sql_custom_sql_expr(params)
File "/tmp/Bazel.runfiles_xzqp1c9m/runfiles/clouddq/clouddq/classes/rule_type.py", line 96, in to_sql_custom_sql_expr
if params.get("rule_binding_arguments", {}).get(argument, None) is None:
**AttributeError: 'list' object has no attribute 'get'**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.