Azure / Azure/Azure-DataFactory

Data Copy Activity does not honor defined schemas

Open
#254 4 comments 3 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

There seems to be a bug in the Data Factory Copy Activity. When we use the partition discovery, the columns that come from the partitions are not following the types defined in the source or sync. They get forced to be UTF8 strings. The Data flows on the other hand does honor the type definitions that we specify in the source and destination. This is leading to parquet files that are generated from Data Copy and Dataflow to be incompatible as the schema is different and there is no way for us to enforce the schema structure in the Data Copy from what I can tell.

Example, on the source and sink we define a column Column1 of type INT32. This column comes from the partition discovery and so in the folder structure contains Column1=#. When we hit import schema in the data copy activity for the conversion, it imports them as String -> UTF8. This leads to the parquet file having a string type as the column. We manually edited the conversion settings to try String->INT32 and the parquet file still had String as the column type. We then tried to do INT32->INT32 and still it was a string. There is no way for us to enforce the right schema in the destination parquet file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.