GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Random Row fails for DECIMAL primary keys with 31 digits
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
**Describe the bug**
Random Row fails for DECIMAL primary keys with 30 digits. The error is likely because these are being cast into floats, and the no rows match due the loss of precision
**Steps to reproduce the behavior**
See that the table has proper data with a large decimal number as primary key.
```
data-validation query -c postgres -q 'select id, col_data from pso_data_validator.dvt_large_decimals'
[(Decimal('123456789012345678901234567890'), 'Row 1'), (Decimal('223456789012345678901234567890'), 'Row 2'), (Decimal('323456789012345678901234567890'), 'Row 3'), (Decimal('423456789012345678901234567890'), 'Row 4'), (Decimal('523456789012345678901234567890'), 'Row 5')]
```
Run validation on this data
```
data-validation validate row -tc postgres -sc postgres -tbls=pso_data_validator.dvt_large_decimals -hash=col_data -rr -rbs=3
╒═══════════════════╤═══════════════════╤═════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════╕
│ validation_name │ validation_type │ source_table_name │ source_column_name │ source_agg_value │ target_agg_value │ pct_difference │ validation_status │ run_id │
╞═══════════════════╪═══════════════════╪═════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════╡
╘═══════════════════╧═══════════════════╧═════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════╛
```
This should show 3 rows and is showing 0 rows, likely because of the loss of precision in converting decimal to float. Also see [random_row_validation.md](docs/random_row_validation.md) for an alternative approach to random row validation.
Sundar Mudupalli
Contributor guide
Assessment
This issue has not been assessed yet.