Azure / Azure/azure-stream-analytics
How can a custom deserializer output a native JSON field (record) ?
- Dominant language
- C#
- Stars
- 229
- Forks
- 920
- PR merge metrics
- No merged PRs in 30d
Description
Here is my situation - I've got an event hub feed that contains escaped JSON in a string, i.e. `"{\"foo\":\"bar\"}"` so I cannot use the built in JSON format. I've managed to build a StreamSerializer based on the examples, but I can only yield back simple structures with string or other primitives as fields. I want to return data that will appear as a native JSON output, just like the built-in one does. Here's what my currently (failing) attempt looks like:
Here's my structure that's `yield`ed back from my StreamSerializer:
```csharp
public class CustomJsonData
{
public string Source { get; set; } = "MySource";
public string JsonString { get; set; } // works fine
public JsonDocument Json { get; set; } // fails
}
```
The serializer itself:
```csharp
public class CustomJsonDeserializer : StreamDeserializer
{
// streamingDiagnostics is used to write error to diagnostic logs
private StreamingDiagnostics streamingDiagnostics;
// Initializes the operator and provides context that is required for publishing diagnostics
public override void Initialize(StreamingContext streamingContext)
{
this.streamingDiagnostics = streamingContext.Diagnostics;
}
// Deserializes a stream into objects of type CustomEvent
public override IEnumerable Deserialize(Stream stream)
{
using (var sr = new StreamReader(stream))
{
string line = line = sr.ReadLine();
JsonDocument json = null;
try {
line = Regex.Unescape(line.Trim('"'));
json = JsonDocument.Parse(line);
} catch (Exception e) {
line = e.Message;
}
yield return new CustomJsonData() {
JsonString = line,
Json = json
};
}
}
}
```
This works fine dumping out a string, but then it's just a string field - we don't get any native JSON support in the Portal UI nor the ability to dot navigate the properties/fields.
So, we _could_ probably use a javascript UDF in the query to ~~`eval` the string~~ JSON.parse() the string but this seems inefficient ~~and less safe~~ than having our serializer emit native JSON.
The JsonDocument above is from System.Text.Json 5.x (to comply with netstandard 2.0) but we get:
```
[Info] 2022-01-26 1:33:45 PM : Job Start Successfully !
[Error] 2022-01-26 1:33:55 PM : **System Exception** Could not load file or assembly 'System.Text.Json, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
[Error] 2022-01-26 1:33:55 PM : at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
[Info] Stopped Local Run.
Child process exited with code 4294967295
[Info] Shutdown local credential server http://localhost:8999/
```
Any tips? Do I need to use a very specific version of System.Text.Json? Do I need a magic attribute somewhere? Do I need to use a differenet base class (i.e. not StreamSerializer?)
Thanks
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.