GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
validate column: Hive/MySQL char(2) column is losing trailing space in column validation
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
Sample Hive column:
```
, col_char_2 char(2)
Sample BigQuery column:
, col_char_2 STRING(2)
COL_CHAR_2 contains A , B , C in both tables, i.e. each value has a trailing space.
```
Ran this validation:
```
data-validation validate column -sc=hive-conn -tc=bq-conn -tbls=test_user.test_table \
--filter-status=fail --max=col_char_2 --sum=col_char_2
```
Output:
```
╒═════════════════════════╤═════════════════╤╤════════════════════╤══════════════════╤══════════════════╤
│ validation_name │ validation_type ││ source_column_name │ source_agg_value │ target_agg_value │
╞═════════════════════════╪═════════════════╪╪════════════════════╪══════════════════╪══════════════════╪
│ max__length__col_char_2 │ Column ││ length__col_char_2 │ 1 │ 2 │
├─────────────────────────┼─────────────────┼┼────────────────────┼──────────────────┼──────────────────┼
│ sum__length__col_char_2 │ Column ││ length__col_char_2 │ 3 │ 6 │
╘═════════════════════════╧═════════════════╧╧════════════════════╧══════════════════╧══════════════════╧
```
The same is true when testing MySQL and Impala to BigQuery validations.
See also: https://github.com/GoogleCloudPlatform/professional-services-data-validator/issues/842
When fixing this we also need to update the integration tests, search for "issue-1514" in `test_hive.py`, `test_impala.py` and `test_mysql.py`.
Contributor guide
Assessment
This issue has not been assessed yet.