Azure / Azure/Azure-DataFactory

DECIMAL conversion in COPY activity chokes on NULLs

Open
#277 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

We're copying data from Netezza to Azure Synapse DWH. Some of the data is represented in various decimal formats but often only has INTEGERS in it or has 4 decimal places in the data when 11 are are available in the column.

If we convert the data types so
DECIAML (38,9) --> INTEGER
or
DECIMAL (38,9) --> DECIMAL (20,4)
we get some "interesting behaviors"

We are using parameterized datasets as we're handling hundreds of tables but the same issue occurs if you map the columns in the copy activity.

If a NULL value is encountered we get some random HIVE CAST DECIMAL error (it really is entirely random, the same row can generate several different errors on repeated runs) even though both columns allow NULL the underlying databricks magic obviously chokes cos who knew a NULL value might have to be handled. The error has the source column datatype/precision in it but no info about the columnn index, column name, or indeed value other than a hint it's a NULL in some of the error messages.

Pulling single rows and or column combinations allows you to track it down but the error messages are obviously not useful (a bit of a pattern for ADF error messages if I might venture an opinion)

Casting the fields as VARCHAR and loading them into VARCHAR then doing conversion inside the Synapse DB seems to be the only reliable method. Which is just shockingly retro.

All in all a bit dire!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.