Azure / Azure/Azure-DataFactory
Error while copying data from sql server to adls gen2 in ORC format
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
I have created a dynamic pipeline for copying data from SQL Server to ADLS Gen 2 in ORC format.
As per the requirement, column names can have special characters (allowed in SQL server) and Parquet and AVRO are not able to handle them, hence ORC format is used. For Example whitespace in column name is not supported in Parquet and AVRO.
An error occurs when the source table has a column name with "," character.
PFB details of error occurred in the pipeline run
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.IndexOutOfBoundsException:Index: 6, Size: 6.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "",
"details": []
}
Since the error message is ambiguous and there is no documentation for ORC format not supporting character "," assuming it a bug.
Sample table schema in SQL server:
DROP TABLE IF EXISTS SAMPLE_DATA
CREATE TABLE SAMPLE_DATA (
[FIRST,MIDDLE NAME] VARCHAR(15)
,[LAST NAME] VARCHAR(15)
,[AGE] INT
)
INSERT INTO SAMPLE_DATA(
[FIRST,MIDDLE NAME]
,[LAST NAME]
,[AGE])
VALUES ('POOJAN RAMESH','KOTHARI',25)

Contributor guide
Assessment
This issue has not been assessed yet.