[TD to BQ] DVT does not account for differences in encoding for primary keys

Hello team,

We have encountered a scenario where a Teradata CHAR column that has special characters is evaluated differently depending on whether or not the column is passed as a primary key.

If the column is not a primary key, the row hash process decodes TD values and translates them to UTF-8 before comparing against the BigQuery value. This allows identical data with different encodings to still pass a validation.

If the column is a primary key, DVT does not alter the value in any way before joining the source and target data together. In this case, the column will be interpreted as two different values in TD and BQ. This results in two distinct rows in the output which both fail, instead of one row that succeeds.

We are exploring custom query options to workaround this issue, but ideally DVT would be able to account for the different encodings in Primary Keys in the same way it accounts for it in a row hash.

Example value: ‘ABCDEFGÂ ‘

The hex code in Teradata is 004100420043004400450046004700C200A0
The hex code in BigQuery is 41424344454647c382c2a0

The key differences here are the following values:
1. "C2" in TD which maps to "c382" in BQ
2. "A0" in TD which maps to "c2a0" in BQ

If this column is not a primary key, the row hash validation succeeds. If the column is a primary key, DVT does not recognize these two rows as the same in the two systems.