GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Column validation: Incorrect failure status for avg on Oracle decimal(38) column

Open
#1,551 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

**Describe the bug**
Note: This bug cannot be actioned until issue 1540 is complete.

A comparison of average validation of 38 digit NUMBER appears to match, `1.23457e+37` from both Oracle and PostgreSQL, but fails:
```
╒═══════════════════╤═══════════════════╤═══════════════════════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════════════════════════════════╕
│ validation_name │ validation_type │ source_table_name │ source_column_name │ source_agg_value │ target_agg_value │ pct_difference │ validation_status │ run_id │
╞═══════════════════╪═══════════════════╪═══════════════════════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════════════════════════════════╡
│ avg__col_dec_38 │ Column │ pso_data_validator.dvt_large_decimals │ col_dec_38 │ 1.23457e+37 │ 1.23457e+37 │ nan │ fail │ 759744b8-fa7c-474e-9bbb-3e3b27d6130e │
╘═══════════════════╧═══════════════════╧═══════════════════════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════════════════════════════════╛
```

Notice pct_difference=`nan`. Also note we get the following warning from the combiner:
```
pandas/core/series.py:2569: RuntimeWarning: overflow encountered in multiply
result = self._values.round(decimals)
```

**What version of DVT are you using?**
7.8 (dev)

**What type of connections are you using for source and target?**
Oracle, PostgreSQL

**Steps to reproduce the behavior**
Test command:
```
data-validation -v validate column -sc=ora -tc=pg -tbls=pso_data_validator.dvt_large_decimals -avg="col_dec_38"
```

**Expected behavior**
Success

Contributor guide

Open the contributing guide

Research direction

Wait for issue 1540, then run the provided data-validation command against the Oracle and PostgreSQL connections and inspect the combiner warning about overflow in pandas Series.round. Done means the avg validation for col_dec_38 reports success with a valid pct_difference rather than nan.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, postgresql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.