GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

validate column: Hive/MySQL char(2) column is losing trailing space in column validation

Open
#1,514 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

Sample Hive column:
```
, col_char_2 char(2)
Sample BigQuery column:

, col_char_2 STRING(2)
COL_CHAR_2 contains A , B , C in both tables, i.e. each value has a trailing space.
```
Ran this validation:
```
data-validation validate column -sc=hive-conn -tc=bq-conn -tbls=test_user.test_table \
--filter-status=fail --max=col_char_2 --sum=col_char_2
```

Output:
```
╒═════════════════════════╤═════════════════╤╤════════════════════╤══════════════════╤══════════════════╤
│ validation_name │ validation_type ││ source_column_name │ source_agg_value │ target_agg_value │
╞═════════════════════════╪═════════════════╪╪════════════════════╪══════════════════╪══════════════════╪
│ max__length__col_char_2 │ Column ││ length__col_char_2 │ 1 │ 2 │
├─────────────────────────┼─────────────────┼┼────────────────────┼──────────────────┼──────────────────┼
│ sum__length__col_char_2 │ Column ││ length__col_char_2 │ 3 │ 6 │
╘═════════════════════════╧═════════════════╧╧════════════════════╧══════════════════╧══════════════════╧
```

The same is true when testing MySQL and Impala to BigQuery validations.

See also: https://github.com/GoogleCloudPlatform/professional-services-data-validator/issues/842
When fixing this we also need to update the integration tests, search for "issue-1514" in `test_hive.py`, `test_impala.py` and `test_mysql.py`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.