dotnet / dotnet/spark

[BUG]: Extension method JsonSerDe SortProperties removes arrays with strings from metadata during serialization

Open
#1,082 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Describe the bug**
If you include an array of values in a StructField metadata instance they will not serialize when .Json is called on the root StructType. They will however show on the fields JsonValue.

**To Reproduce**

Steps to reproduce the behavior:
```
var values = new[] { "Test1", "Test2" };
var metadata = new JObject(new JProperty("allowedValues", values));
var field = new StructField("TestField", new StringType(), true, metadata);
var fields = new StructType(new[] { field });
var json = fields.Json;
```
json will contain `{"fields":[{"metadata":{"allowedValues":[]},"name":"TestField","nullable":true,"type":"string"}],"type":"struct"}`

**Expected behavior**
I would expect allowedValues in the example above to contain Test1 and Test2.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Microsoft.Spark 2.1.1

**Additional context**
The code that sorts the array in the serialization when you call DataType.Json does not take into account if the item is a simple string and thus returns an empty array:
https://github.com/dotnet/spark/blob/45d0058e315825c905996e87dcaf8bc0feeb64f0/src/csharp/Microsoft.Spark/Interop/Ipc/JsonSerDe.cs#L55-L65

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.