System.OutOfMemoryException in AvroExtractor
- Dominant language
- C#
- Stars
- 231
- Forks
- 666
- PR merge metrics
- No merged PRs in 30d
Description
I have a 320mb big avro file. When using the AvroExtractor on it I get a `System.OutOfMemoryException`.
```
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor.d__3.MoveNext()
at ScopeEngine.SqlIpExtractor.GetNextRow(SqlIpExtractor* , Extract_2_Data0* output) in d:\data\ccs\jobs\3b349459-c713-4500-b28b-3ecc540f25b5_v0\sqlmanaged.h:line 1924
```
The docs here https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-programmability-guide says that the limit is 0.5GB for a UDO. I guess it make sense with the exception, the extractor copies the input stream which is 320mb.
I dont have full control of the input avro file, it is created from a Stream Analytics Jobs that reads from a event hub partitioned by partitionid, and then save the result to blob which is read by the u-sql job.
How can I work around this problem?
Contributor guide
Assessment
This issue has not been assessed yet.