GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Random Row fails for DECIMAL primary keys with 31 digits

Open
#1,822 1 comment 0 reactions 1 assignee Claimed by @nj1973 View on GitHub
type: bug
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

**Describe the bug**
Random Row fails for DECIMAL primary keys with 30 digits. The error is likely because these are being cast into floats, and the no rows match due the loss of precision

**Steps to reproduce the behavior**
See that the table has proper data with a large decimal number as primary key.
```
data-validation query -c postgres -q 'select id, col_data from pso_data_validator.dvt_large_decimals'
[(Decimal('123456789012345678901234567890'), 'Row 1'), (Decimal('223456789012345678901234567890'), 'Row 2'), (Decimal('323456789012345678901234567890'), 'Row 3'), (Decimal('423456789012345678901234567890'), 'Row 4'), (Decimal('523456789012345678901234567890'), 'Row 5')]
```
Run validation on this data
```
data-validation validate row -tc postgres -sc postgres -tbls=pso_data_validator.dvt_large_decimals -hash=col_data -rr -rbs=3
╒═══════════════════╤═══════════════════╤═════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════╕
│ validation_name │ validation_type │ source_table_name │ source_column_name │ source_agg_value │ target_agg_value │ pct_difference │ validation_status │ run_id │
╞═══════════════════╪═══════════════════╪═════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════╡
╘═══════════════════╧═══════════════════╧═════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════╛
```

This should show 3 rows and is showing 0 rows, likely because of the loss of precision in converting decimal to float. Also see [random_row_validation.md](docs/random_row_validation.md) for an alternative approach to random row validation.

Sundar Mudupalli

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.