Azure / Azure/usql

Job fails when trying to insert large number of rows to table with large number of columns. “A commit request is outside of a reservation size. A row is too big.”

Open
#123 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
231
Forks
666
PR merge metrics
No merged PRs in 30d

Description

I am quite new to Azure Data Lake and U-SQL and have worked with SQL earlier. I am facing the following issue while trying to migrate a SQL application to cloud.

I have ADLA table with 73 columns. I need to populate it with data in 3 columns and null in other columns. The data comes after processing 2 other tables. From error message, I understood that one particular row size is too big and checked the data by outputting in CSV. Each row is 3 columns (long?, int and int?) and nulls in 70 other columns. The data is inside the long and int range so there is no data type/size issue.

Observations:
• When I tried to insert large number of rows in a table with large number of column, It failed. (approx. 200,000 rows and table with 73 columns) error message: **“A commit request is outside of a reservation size. A row is too big.”(figure 1)**
• When I tried to insert the same data in a separate table with only 3 columns I am interested in, it passes.
• When I tried to insert the same rows in batches (I tried sorting data and inserting half in one ADLA job and other half in another ADLA job), it succeeded. (I think that there is no data size restriction in ADLA table but there is some restriction while processing it)**(figure 2)**
• When I tried to output the processed rows directly to CSV in stead of inserting into table, it succeeds without any restriction. I can output all the data at once. (I think there is no problem with data types or fetching/joining logic)**(figure 3)**
• I think there is limitation in processing large amount of data if it is to be inserted into ADLA Table with large number of columns and this limitation does not exist while creating files instead of inserting to tables.

Is there something I have missed taking into consideration? How can I resolve the issue?

![image](https://user-images.githubusercontent.com/975334/36892368-8722bafa-1e2a-11e8-8a1d-fa748dff97f6.png) ![image](https://user-images.githubusercontent.com/975334/36892387-9708ecd2-1e2a-11e8-8afb-bd08d918bc2c.png) ![image](https://user-images.githubusercontent.com/975334/36892406-a1bff8b4-1e2a-11e8-8e26-696879d9d5a7.png)

I recently found in forums that size of string is 128KB and read [here](https://msdn.microsoft.com/en-us/azure/data-lake-analytics/u-sql/rowset-the-processing-data-structure-u-sql) that each row can be upto 4 MB.
I have table with 48 string columns which makes it 6144 KB = 6MB. Is this the problem I am facing? If so, is there any plans to increase data size per row? Is it generally a bad practice to have such large columns?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.