Avro Extractor Not Working
- Dominant language
- C#
- Stars
- 231
- Forks
- 666
- PR merge metrics
- No merged PRs in 30d
Description
After doing all the steps described in your repository for reading the Avro file from Data Lake Store using U-SQL.
Below is the script I am using which is replication of your sample script,also the data used here is the same provided by you.But after submitting the job I get this error:
`Use 'IUnstructuredReader.Length' instead. For more information, see documentation about 'IUnstructuredReader.Length' and 'IUnstructuredReader.BaseStream'.`
Can anyone tell me how to resolve this out.I don't wanna change the internal code of the DLL's used here.
`/*DROP ASSEMBLY IF EXISTS [Avro];
CREATE ASSEMBLY [Avro] FROM @"/assemblies/Avro.dll";
DROP ASSEMBLY IF EXISTS [Microsoft.Analytics.Samples.Formats];
CREATE ASSEMBLY [Microsoft.Analytics.Samples.Formats] FROM @"/assemblies/Microsoft.Analytics.Samples.Formats.dll";
DROP ASSEMBLY IF EXISTS [Newtonsoft.Json];
CREATE ASSEMBLY [Newtonsoft.Json] FROM @"/assemblies/Newtonsoft.Json.dll";
DROP ASSEMBLY IF EXISTS [log4net];
CREATE ASSEMBLY [log4net] FROM @"/assemblies/log4net.dll";*/
REFERENCE ASSEMBLY [Newtonsoft.Json];
REFERENCE ASSEMBLY [log4net];
REFERENCE ASSEMBLY [Avro];
REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];
REFERENCE ASSEMBLY [Microsoft.Hadoop.Avro];
USING Microsoft.Analytics.Samples.Formats.Avro;
USING Microsoft.Analytics.Samples.Formats.Json;
DECLARE @input_file string = @"/sample-data/twitter.avro";
DECLARE @output_file string = @"/sample-data/abc.csv";
@rs =
EXTRACT
createdat string,
topic string,
sentimentscore long,
eventprocessedutctime string,
partitionid long,
eventenqueuedutctime string
FROM @input_file
USING new AvroExtractor(@"
{
""type"" : ""record"",
""name"" : ""GenericFromIRecord0"",
""namespace"" : ""Microsoft.Streaming.Avro"",
""fields"" : [ {
""name"" : ""createdat"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""topic"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""sentimentscore"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventprocessedutctime"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""partitionid"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventenqueuedutctime"",
""type"" : [ ""null"", ""string"" ]
} ]
}
");
@cnt =
SELECT topic, COUNT(*) AS cnt
FROM @rs
GROUP BY topic;
OUTPUT @cnt TO @output_file USING Outputters.Text();`
Contributor guide
Assessment
This issue has not been assessed yet.