GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

JSON formatted output requires post-processing

Open
#1,394 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

When requesting JSON output via `--format=json` the obvious JSON stricture would be include an array containing the validation results. Instead what we get is a map keyed on the row number. For example:

```
data-validation validate column -sc ora -tc pg --tables-list=pso_data_validator.dvt_core_types --sum="id" --format=json
{
"0":{"validation_name":"count","validation_type":"Column","aggregation_type":"count","source_table_name":"pso_data_validator.dvt_core_types","source_column_name":null,"source_agg_value":"3","target_table_name":"pso_data_validator.dvt_core_types","target_column_name":null,"target_agg_value":"3","group_by_columns":null,"primary_keys":null,"num_random_rows":null,"difference":0.0,"pct_difference":0.0,"pct_threshold":0.0,"validation_status":"success","run_id":"3ba9b1ea-9192-4301-aa4d-070260175922","labels":[],"start_time":1736775679472,"end_time":1736775680025},
"1":{"validation_name":"sum__id","validation_type":"Column","aggregation_type":"sum","source_table_name":"pso_data_validator.dvt_core_types","source_column_name":"id","source_agg_value":"6","target_table_name":"pso_data_validator.dvt_core_types","target_column_name":"id","target_agg_value":"6","group_by_columns":null,"primary_keys":null,"num_random_rows":null,"difference":0.0,"pct_difference":0.0,"pct_threshold":0.0,"validation_status":"success","run_id":"3ba9b1ea-9192-4301-aa4d-070260175922","labels":[],"start_time":1736775679472,"end_time":1736775680025}
}
```

The customer expected an array as below:
```
[
{"validation_name":"count","validation_type":"Column","aggregation_type":"count","source_table_name":"pso_data_validator.dvt_core_types","source_column_name":null,"source_agg_value":"3","target_table_name":"pso_data_validator.dvt_core_types","target_column_name":null,"target_agg_value":"3","group_by_columns":null,"primary_keys":null,"num_random_rows":null,"difference":0.0,"pct_difference":0.0,"pct_threshold":0.0,"validation_status":"success","run_id":"3ba9b1ea-9192-4301-aa4d-070260175922","labels":[],"start_time":1736775679472,"end_time":1736775680025},
{"validation_name":"sum__id","validation_type":"Column","aggregation_type":"sum","source_table_name":"pso_data_validator.dvt_core_types","source_column_name":"id","source_agg_value":"6","target_table_name":"pso_data_validator.dvt_core_types","target_column_name":"id","target_agg_value":"6","group_by_columns":null,"primary_keys":null,"num_random_rows":null,"difference":0.0,"pct_difference":0.0,"pct_threshold":0.0,"validation_status":"success","run_id":"3ba9b1ea-9192-4301-aa4d-070260175922","labels":[],"start_time":1736775679472,"end_time":1736775680025}
]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.