GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

[Question] Possibilty of Multiple Filter Setup for List of Tables

Open
#1,526 1 comment 0 reactions 1 assignee Assigned to @nehanene15 View on GitHub
type: question
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

Hi everyone!

**Background**
I have a use case where I need to run column level validations against 100s of BigQuery sources & targets every day. The validations would be broken up into smaller chunks of ~20 tables per `data-validation` invocation.

The tables that need to be validated are very large, and since we're in BigQuery, this means costs could be sky high. Thankfully, we can use the `--filters` flag in our `data-validation` command to filter partition columns (TIMESTAMP) to reduce cost.

**Issue**
While the partition column is the same for all tables, not all tables contain the same date range of data. So, a simple universal filter like `TIMESTAMP in (random_days)` would not provide proper validation coverage over all tables.

**Question**
Is it possible to have comma-separated values in `--filters` and have them map (by position) to a table in `tbls`?

For example, take the following command. We are performing validation on `dataset_one.table_one` & `dataset_two.table_two`. We have also defined two filters in `--filters` separated by a comma.

This means that when the validation is run, the 1st filter is applied to the 1st table, and the 2nd filter is applied to the 2nd table.

Is this (or anything similar to this) supported by `data-validation`?

```
data-validation validate column \
-sc bigquery-conn1 \
-tc bigquery-conn2 \
-tbls "dataset_one.table_one, dataset_two.table_two" \
--sum "*" \
--filters "insert_time='2025-01-12', insert_time='2021-05-30"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.