Azure / Azure/Azure-DataFactory

Preview and import schema fail on Avro dataset without compression

Open
#158 4 comments 0 reactions 1 assignee Claimed by @DayhopeZhang View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

**Context:**
I'm working on a process to import and transform some data into Table Storage and want to use Avro as my intermediate format between my mapping flow and the copy action since I'm storing some binary format data in a column.
Trying to use JSON doesn't work well as the data format won't be preserved and manually specifying the datatype as binary or byte[] in the copy action mapping will fail.
I've tried with Parquet and this works fine, but is very slow in comparison as this is a column based file format while I'm doing all row based operations (~2s vs ~500ms on a 20.000 row test set).
So Avro will save quite some processing time when we'll process the full set with millions of rows).

**Problem:**
When selecting Avro as file format the mapping flow completes succesfully. But when trying to preview the data or import the schema in setting up the copy activity I get the following error:
"Avro only supports deflate compression while 'snappy' is not supported. Codec name 'snappy' is unknown. Parameter name: codecName . Activity ID: {REDACTED}"
![image](https://user-images.githubusercontent.com/4289336/74426886-7a0c4580-4e56-11ea-8c92-37d3354e1fdf.png)

But I haven't set compression on the dataset:
![image](https://user-images.githubusercontent.com/4289336/74425715-6b249380-4e54-11ea-8093-40a39e7da5f8.png)

It seems the preview and import options try to use the snappy compression, while no compression is set. And snappy not even being supported (although it can be selected on the dataset in the UI).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.