GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Enable data-validation validate custom-query row with only one query file

Open
#1,101 1 comment 0 reactions 0 assignees View on GitHub
priority: p2
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

Currently both `--source-query-file` and `--target-query-file` are bot required to run a custom query row validation, i.e. `validate custom-query row`.

It might be the case that when migrating a table from source to target, the target version has a change that can be accounted for by a custom SQL on only one of the sides (source or target), without requiring two custom SQL.

For example, it is particularly common in BQ migrations to replace surrogate keys with composite natural keys. To do row comparison a custom SQL is needed to compute the composite natural **in the source**, therefore a `--source-query-file` is needed. Nevertheless the result from this custom SQL can be compared to the target table, thus the `--target-query-file` is not needed.

The easy workaround is to use a `SELECT *` in the `--target-query-file`, which brings to mind:
1. Having a `SELECT * ...` as a custom SQL is evidence enough that the custom sql is not needed
2. I would expect that performing a `SELECT * ...` would incur in additional computational costs compared to accessing the table directly, specially in the case of random row validation

If (2) holds true, I believe this request to be relevant. If not this is just a nice to have

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.