Azure / Azure/usql

Avro Extractor Not Working

Open
#122 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
231
Forks
666
PR merge metrics
No merged PRs in 30d

Description

After doing all the steps described in your repository for reading the Avro file from Data Lake Store using U-SQL.
Below is the script I am using which is replication of your sample script,also the data used here is the same provided by you.But after submitting the job I get this error:

`Use 'IUnstructuredReader.Length' instead. For more information, see documentation about 'IUnstructuredReader.Length' and 'IUnstructuredReader.BaseStream'.`

Can anyone tell me how to resolve this out.I don't wanna change the internal code of the DLL's used here.

`/*DROP ASSEMBLY IF EXISTS [Avro];
CREATE ASSEMBLY [Avro] FROM @"/assemblies/Avro.dll";
DROP ASSEMBLY IF EXISTS [Microsoft.Analytics.Samples.Formats];
CREATE ASSEMBLY [Microsoft.Analytics.Samples.Formats] FROM @"/assemblies/Microsoft.Analytics.Samples.Formats.dll";
DROP ASSEMBLY IF EXISTS [Newtonsoft.Json];
CREATE ASSEMBLY [Newtonsoft.Json] FROM @"/assemblies/Newtonsoft.Json.dll";
DROP ASSEMBLY IF EXISTS [log4net];
CREATE ASSEMBLY [log4net] FROM @"/assemblies/log4net.dll";*/

REFERENCE ASSEMBLY [Newtonsoft.Json];
REFERENCE ASSEMBLY [log4net];
REFERENCE ASSEMBLY [Avro];
REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];
REFERENCE ASSEMBLY [Microsoft.Hadoop.Avro];

USING Microsoft.Analytics.Samples.Formats.Avro;
USING Microsoft.Analytics.Samples.Formats.Json;

DECLARE @input_file string = @"/sample-data/twitter.avro";
DECLARE @output_file string = @"/sample-data/abc.csv";

@rs =
EXTRACT
createdat string,
topic string,
sentimentscore long,
eventprocessedutctime string,
partitionid long,
eventenqueuedutctime string
FROM @input_file
USING new AvroExtractor(@"
{
""type"" : ""record"",
""name"" : ""GenericFromIRecord0"",
""namespace"" : ""Microsoft.Streaming.Avro"",
""fields"" : [ {
""name"" : ""createdat"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""topic"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""sentimentscore"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventprocessedutctime"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""partitionid"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventenqueuedutctime"",
""type"" : [ ""null"", ""string"" ]
} ]
}
");

@cnt =
SELECT topic, COUNT(*) AS cnt
FROM @rs
GROUP BY topic;

OUTPUT @cnt TO @output_file USING Outputters.Text();`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.