GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Option to continue on error when validation command has multiple input tables
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
**Is your feature request related to a problem? Please describe.**
We run DVT commands on the output of find-tables, for example:
```
data-validation validate column -sc=ora -tc=pg \
-tbls="$(data-validation find-tables -sc=ora -tc --allowed-schemas pso_data_validator)" \
--count="*" --sum="*" --min="*" --max="*"
```
There might be 100s of tables and if the third one throws an exception the remaining 97 are not executed.
Similarly we wrote validations across 10 YAML files so we can run in parallel. If one of the early validations in a file throws an exception then the remainder of the validations are not executed.
**What version of DVT are you using?**
8.7.x
**What type of connections are you using for source and target?**
N/A
**Describe the solution you'd like**
We would like a new option called something like `--continue-on-exception` that instructs DVT to continue running subsequent validations even when one fails. Maybe with a logged message identifying which validation threw the error.
**Describe alternatives you've considered**
We've had to break find-tables output up into single tables and create a shell script which runs each table individually. Therefore we have a workaround but would prefer DVT to be able to continue past exceptions.
**How are you executing DVT?**
CLI commands on VM and github workflows.
Contributor guide
Research direction
Start at the CLI validation command entry point and trace how validation exceptions stop processing across multiple tables and YAML files. Define the new continue-on-exception behavior, including identifying logged failures, and verify that later validations still run after an earlier exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100