GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
validate column casts sum of integer column to STRING, preventing percentage difference calculation
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
**Describe the bug**
When using the validate column command with the sum aggregation on an integer column, the Data Validation Tool automatically casts the result of the sum to STRING. This prevents the tool from calculating the pct_difference, resulting in a null value in the results.
**What version of DVT are you using?**
7.5.2
**What type of connections are you using for source and target?**
BigQuery
**Steps to reproduce the behavior**
1.Create Source and Target Tables in BigQuery:
CREATE OR REPLACE TABLE `gcp-dev.gcp_temp.sample_source_tbl` (
id INT64,
name STRING,
dept STRING,
sal INT64,
doj DATE,
modifydt DATE
);
CREATE OR REPLACE TABLE `gcp-dev.gcp_temp.sample_target_tbl_2` (
id INT64,
name STRING,
dept STRING,
sal INT64,
doj DATE,
modifydt DATE
);
Insert Sample Data:
INSERT INTO `gcp-dev.gcp_temp.sample_source_tbl`
VALUES
(01, 'name01', 'dept01', 100, '2024-01-01', '2024-01-01'),
(02, 'name02', 'dept02', 200, '2024-01-02', '2024-01-01'),
(03, 'name03', 'dept03', 300, '2024-01-03', '2024-01-01'),
(04, 'name04', 'dept04', 400, '2024-01-04', '2024-01-01'),
(05, 'name05', 'dept05', 500, '2024-01-05', '2024-01-01'),
(06, 'name06', 'dept06', 600, '2024-01-06', '2024-01-02'),
(07, 'name07', 'dept07', 700, '2024-01-07', '2024-01-02'),
(08, 'name08', 'dept08', 800, '2024-01-08', '2024-01-02'),
(09, 'name09', 'dept09', 900, '2024-01-09', '2024-01-02'),
(10, 'name10', 'dept10', 1000, '2024-01-10', '2024-01-02');
INSERT INTO `gcp-dev.gcp_temp.sample_target_tbl_2`
VALUES
(02, 'name02a', 'dept01', 100, '2024-01-01', '2024-01-02'),
(02, 'name02', 'dept02', 200, '2024-01-02', '2024-01-01'),
(03, 'name03', 'dept03', 300, '2024-01-03', '2024-01-02'),
(04, 'name04', 'dept04', 400, '2024-01-04', '2024-01-01'),
(05, 'name05', 'dept05a', 500, '2024-01-05', '2024-01-01'),
(06, 'name06', 'dept06', 600, '2024-01-06', '2024-01-02'),
(07, 'name07', 'dept07', 700, '2024-01-07', '2024-01-01'),
(08, 'name08', 'dept08', 800, '2024-01-08', '2024-01-02'),
(09, 'name09', 'dept09', 901, '2024-01-09', '2024-01-02'),
(10, 'name10', 'dept10', 1000, '2024-01-10', '2024-01-02');
Run the Data Validation Tool:
2. Execute
# Run command
data-validation --verbose validate column \
--source-conn my_bq_conn \
--target-conn my_bq_conn \
--tables-list gcp_temp.sample_source_tbl=gcp_temp.sample_target_tbl_2 \
--sum sal \
--result-handler my_bq_conn.results_table \
--labels env=dev,purpose=data_migration,run_date=$(date +%Y-%m-%d)
Replace my_bq_conn and results_table with your actual values.
4. Examine Results:
Query the results_table. The pct_difference column for the sum__sal validation will be null.
5. Expected behavior:
The pct_difference column should contain the percentage difference between the sum of the sal column in the source and target tables.
Actual Results (from the results_table):
{
"run_id": "your_run_id",
"validation_name": "sum__sal",
"validation_type": "Column",
// ... other fields ...
"source_agg_value": "5500",
"target_agg_value": "5501",
"difference": null,
"pct_difference": null,
"pct_threshold": "0.0",
"validation_status": "fail",
// ... other fields ...
}
Expected Results (if the issue were fixed):
{
"run_id": "your_run_id",
"validation_name": "sum__sal",
"validation_type": "Column",
// ... other fields ...
"source_agg_value": "5500",
"target_agg_value": "5501",
"difference": "1",
"pct_difference": "0.01818181818181818", // or a similar value
"pct_threshold": "0.0",
"validation_status": "fail",
// ... other fields ...
}
**Screenshots**
Screenshots/logs:
04/06/2025 11:32:04 AM-INFO: {'data_client': , 'schema_name': 'gcp_temp', 'table_name': 'sample_source_tbl', 'source_query': None}
04/06/2025 11:32:04 AM-INFO: -- ** Source Query ** --
04/06/2025 11:32:04 AM-INFO: SELECT count(1) AS `count`, count(t0.`id`) AS `count__id`,
count(t0.`length__name`) AS `count__length__name`,
count(t0.`length__dept`) AS `count__length__dept`,
count(t0.`sal`) AS `count__sal`, count(t0.`doj`) AS `count__doj`,
count(t0.`modifydt`) AS `count__modifydt`,
CAST(sum(t0.`id`) AS STRING) AS `sum__id`,
CAST(sum(t0.`sal`) AS STRING) AS `sum__sal`,
avg(t0.`id`) AS `avg__id`, avg(t0.`sal`) AS `avg__sal`,
min(t0.`id`) AS `min__id`, min(t0.`sal`) AS `min__sal`,
max(t0.`id`) AS `max__id`, max(t0.`sal`) AS `max__sal`
FROM (
SELECT t1.*, length(t1.`name`) AS `length__name`,
length(t1.`dept`) AS `length__dept`
FROM `gcp-dev.gcp_temp.sample_source_tbl` t1
) t0
04/06/2025 11:32:07 AM-INFO: {'data_client': , 'schema_name': 'gcp_temp', 'table_name': 'sample_target_tbl_2', 'target_query': None}
04/06/2025 11:32:07 AM-INFO: -- ** Target Query ** --
04/06/2025 11:32:07 AM-INFO: SELECT count(1) AS `count`, count(t0.`id`) AS `count__id`,
count(t0.`length__name`) AS `count__length__name`,
count(t0.`length__dept`) AS `count__length__dept`,
count(t0.`sal`) AS `count__sal`, count(t0.`doj`) AS `count__doj`,
count(t0.`modifydt`) AS `count__modifydt`,
CAST(sum(t0.`id`) AS STRING) AS `sum__id`,
CAST(sum(t0.`sal`) AS STRING) AS `sum__sal`,
avg(t0.`id`) AS `avg__id`, avg(t0.`sal`) AS `avg__sal`,
min(t0.`id`) AS `min__id`, min(t0.`sal`) AS `min__sal`,
max(t0.`id`) AS `max__id`, max(t0.`sal`) AS `max__sal`
FROM (
SELECT t1.*, length(t1.`name`) AS `length__name`,
length(t1.`dept`) AS `length__dept`
FROM `gcp-dev.gcp_temp.sample_target_tbl_2` t1
) t0
04/06/2025 11:32:13 AM-INFO: Results written to BigQuery, run id: afa5cf18-613b-4f93-
**How are you executing DVT?**
CLI commands on local computer
**Additional context**
Basic validations are failing due to this abnormal behavior of the DVT Tool
Contributor guide
Assessment
This issue has not been assessed yet.