GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Ensure consistent time zones for TIME fields in row hash validations

Open
#1,301 3 comments 3 reactions 0 assignees View on GitHub
type: question
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

DVT assumes BQ time fields are in UTC, and only casts time on the Teradata end to GMT to account for this offset. With our customer, we have found this is not always the case. For example when data is loaded to BQ from external source systems, we are seeing data in BQ with EST time zones.

As a result, casting to GMT only on the TD side hashes creates instances where two identical time fields fail a row hash and two very different time values can potentially succeed.

This also creates an inconsistency between the row hash and compare fields validation, as comp fields will pass validation without modifying the timezones.

Is there any way to ensure the casting to a timezone is consistent between both TD and BQ, to avoid false positive/negative validations in the row hash?

Thanks very much!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.