Azure / Azure/Azure-DataFactory

Error while copying data from sql server to adls gen2 in ORC format

Open
#166 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I have created a dynamic pipeline for copying data from SQL Server to ADLS Gen 2 in ORC format.

As per the requirement, column names can have special characters (allowed in SQL server) and Parquet and AVRO are not able to handle them, hence ORC format is used. For Example whitespace in column name is not supported in Parquet and AVRO.

An error occurs when the source table has a column name with "," character.

PFB details of error occurred in the pipeline run
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.IndexOutOfBoundsException:Index: 6, Size: 6.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "",
"details": []
}

Since the error message is ambiguous and there is no documentation for ORC format not supporting character "," assuming it a bug.

Sample table schema in SQL server:

DROP TABLE IF EXISTS SAMPLE_DATA

CREATE TABLE SAMPLE_DATA (
[FIRST,MIDDLE NAME] VARCHAR(15)
,[LAST NAME] VARCHAR(15)
,[AGE] INT
)

INSERT INTO SAMPLE_DATA(
[FIRST,MIDDLE NAME]
,[LAST NAME]
,[AGE])
VALUES ('POOJAN RAMESH','KOTHARI',25)
![image](https://user-images.githubusercontent.com/52167271/76930815-5fbdff80-690d-11ea-9fcb-858529441a93.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.