confluentinc / confluentinc/cli

CLI-3553: Avro JSON Encoding Union Type Fails to Decode

Open
#2,822 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
80
Forks
35
Avg merge
2d 10h
Merged PRs (30d)
33

Description

When attempting to write records to Confluent Cloud using the CLI, we sometimes get errors related JSON Encoded Avro records that deal with union types.

According to the [JSON Encoding spec from Avro](https://avro.apache.org/docs/1.11.1/specification/#json-encoding), when using a union type

> * if its type is null, then it is encoded as a JSON null;
> * otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.

We can do this locally using the included `/usr/bin/kafka-avro-console-producer` inside of the CP Schema Registry docker container, but not using the CLI. Instead, we get

```
Error: cannot decode textual record ".": cannot decode textual union: cannot decode textual map: cannot determine codec: "" for key: ""
```

Below is the full schema (with some sensitive bits removed) and the sample record we are using

```
{
"type" : "record",
"name" : "Updated",
"namespace" : "namespace",
"doc" : "Updated Event",
"fields" : [ {
"name" : "subjectivity_id",
"type" : {
"type" : "string",
"logicalType" : "string"
},
"doc" : "Identifier"
}, {
"name" : "topic",
"type" : "string",
"doc" : "Topic name"
}, {
"name" : "created_at",
"type" : [ "null", {
"type" : "long",
"logicalType" : "timestamp-millis"
} ],
"doc" : "Date indicating when the record was created",
"default" : null
}, {
"name" : "updated_at",
"type" : [ "null", {
"type" : "long",
"logicalType" : "timestamp-millis"
} ],
"doc" : "Date indicating when the record was updated",
"default" : null
}]
}
```

```json
{ "subjectivity_id": "subj-1234","topic": "subjectivity_updated","created_at": null,"updated_at": { "long": 1696613296 }}
```

Let me know what other data we can provide!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.