Azure / Azure/Azure-DataFactory

ORC files don't behave the same way as PARQUET files when importing into AzureSynapse

Open
#276 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

We've had some issues using Polybase (CRLF chars and long text columns) so we have been using ORC files and PARQUET files as an intermediate format (2 copy activities, one dumps out into ORC or PARQUET then another to copy the file content into Azure Synapse DWH instance)

This is a conventional DWH data import so the dest table has a lot of audit columns (INSERT_TIMESTAMP, PROCESS_ID etc.) that are set to allow NULL (We update them afterwards with the values we need)

The inconsistent behavior is when we use PARQUET the data imports and the audit columns are NULL (as expected) when we import via ORC it chokes because the number of columns in the file must be the same as the number of destination columns (we're not using mappings cos we're importing hundreds of tables so we use parametrized data sest)

So the ORC import insists on having values for every column (very Polybase of it) whereas the PARQUET doesn't. Twiddling the "use type default" flag makes no difference and the behavior is the same regardless of which copy method is selected (Polybase, Copy command or Bulk Insert)

All in all a somewhat frustrating and entirely random behavior that really should be more consistent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.