GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
[Question] Possibilty of Multiple Filter Setup for List of Tables
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
Hi everyone!
**Background**
I have a use case where I need to run column level validations against 100s of BigQuery sources & targets every day. The validations would be broken up into smaller chunks of ~20 tables per `data-validation` invocation.
The tables that need to be validated are very large, and since we're in BigQuery, this means costs could be sky high. Thankfully, we can use the `--filters` flag in our `data-validation` command to filter partition columns (TIMESTAMP) to reduce cost.
**Issue**
While the partition column is the same for all tables, not all tables contain the same date range of data. So, a simple universal filter like `TIMESTAMP in (random_days)` would not provide proper validation coverage over all tables.
**Question**
Is it possible to have comma-separated values in `--filters` and have them map (by position) to a table in `tbls`?
For example, take the following command. We are performing validation on `dataset_one.table_one` & `dataset_two.table_two`. We have also defined two filters in `--filters` separated by a comma.
This means that when the validation is run, the 1st filter is applied to the 1st table, and the 2nd filter is applied to the 2nd table.
Is this (or anything similar to this) supported by `data-validation`?
```
data-validation validate column \
-sc bigquery-conn1 \
-tc bigquery-conn2 \
-tbls "dataset_one.table_one, dataset_two.table_two" \
--sum "*" \
--filters "insert_time='2025-01-12', insert_time='2021-05-30"
```
Contributor guide
Assessment
This issue has not been assessed yet.