Azure / Azure/Azure-DataFactory
ORC files don't behave the same way as PARQUET files when importing into AzureSynapse
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
We've had some issues using Polybase (CRLF chars and long text columns) so we have been using ORC files and PARQUET files as an intermediate format (2 copy activities, one dumps out into ORC or PARQUET then another to copy the file content into Azure Synapse DWH instance)
This is a conventional DWH data import so the dest table has a lot of audit columns (INSERT_TIMESTAMP, PROCESS_ID etc.) that are set to allow NULL (We update them afterwards with the values we need)
The inconsistent behavior is when we use PARQUET the data imports and the audit columns are NULL (as expected) when we import via ORC it chokes because the number of columns in the file must be the same as the number of destination columns (we're not using mappings cos we're importing hundreds of tables so we use parametrized data sest)
So the ORC import insists on having values for every column (very Polybase of it) whereas the PARQUET doesn't. Twiddling the "use type default" flag makes no difference and the behavior is the same regardless of which copy method is selected (Polybase, Copy command or Bulk Insert)
All in all a somewhat frustrating and entirely random behavior that really should be more consistent.
Contributor guide
Assessment
This issue has not been assessed yet.