GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Column validation: Incorrect failure status for avg on Oracle decimal(38) column
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
**Describe the bug**
Note: This bug cannot be actioned until issue 1540 is complete.
A comparison of average validation of 38 digit NUMBER appears to match, `1.23457e+37` from both Oracle and PostgreSQL, but fails:
```
╒═══════════════════╤═══════════════════╤═══════════════════════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════════════════════════════════╕
│ validation_name │ validation_type │ source_table_name │ source_column_name │ source_agg_value │ target_agg_value │ pct_difference │ validation_status │ run_id │
╞═══════════════════╪═══════════════════╪═══════════════════════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════════════════════════════════╡
│ avg__col_dec_38 │ Column │ pso_data_validator.dvt_large_decimals │ col_dec_38 │ 1.23457e+37 │ 1.23457e+37 │ nan │ fail │ 759744b8-fa7c-474e-9bbb-3e3b27d6130e │
╘═══════════════════╧═══════════════════╧═══════════════════════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════════════════════════════════╛
```
Notice pct_difference=`nan`. Also note we get the following warning from the combiner:
```
pandas/core/series.py:2569: RuntimeWarning: overflow encountered in multiply
result = self._values.round(decimals)
```
**What version of DVT are you using?**
7.8 (dev)
**What type of connections are you using for source and target?**
Oracle, PostgreSQL
**Steps to reproduce the behavior**
Test command:
```
data-validation -v validate column -sc=ora -tc=pg -tbls=pso_data_validator.dvt_large_decimals -avg="col_dec_38"
```
**Expected behavior**
Success
Contributor guide
Research direction
Wait for issue 1540, then run the provided data-validation command against the Oracle and PostgreSQL connections and inspect the combiner warning about overflow in pandas Series.round. Done means the avg validation for col_dec_38 reports success with a valid pct_difference rather than nan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100