Azure / Azure/usql

System.OutOfMemoryException in AvroExtractor

Open
#107 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
231
Forks
666
PR merge metrics
No merged PRs in 30d

Description

I have a 320mb big avro file. When using the AvroExtractor on it I get a `System.OutOfMemoryException`.

```
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor.d__3.MoveNext()
at ScopeEngine.SqlIpExtractor.GetNextRow(SqlIpExtractor* , Extract_2_Data0* output) in d:\data\ccs\jobs\3b349459-c713-4500-b28b-3ecc540f25b5_v0\sqlmanaged.h:line 1924
```

The docs here https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-programmability-guide says that the limit is 0.5GB for a UDO. I guess it make sense with the exception, the extractor copies the input stream which is 320mb.

I dont have full control of the input avro file, it is created from a Stream Analytics Jobs that reads from a event hub partitioned by partitionid, and then save the result to blob which is read by the u-sql job.

How can I work around this problem?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.