confluentinc / confluentinc/cli
CLI-3553: Avro JSON Encoding Union Type Fails to Decode
- Dominant language
- Go
- Stars
- 80
- Forks
- 35
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 33
Description
When attempting to write records to Confluent Cloud using the CLI, we sometimes get errors related JSON Encoded Avro records that deal with union types.
According to the [JSON Encoding spec from Avro](https://avro.apache.org/docs/1.11.1/specification/#json-encoding), when using a union type
> * if its type is null, then it is encoded as a JSON null;
> * otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.
We can do this locally using the included `/usr/bin/kafka-avro-console-producer` inside of the CP Schema Registry docker container, but not using the CLI. Instead, we get
```
Error: cannot decode textual record ".": cannot decode textual union: cannot decode textual map: cannot determine codec: "" for key: ""
```
Below is the full schema (with some sensitive bits removed) and the sample record we are using
```
{
"type" : "record",
"name" : "Updated",
"namespace" : "namespace",
"doc" : "Updated Event",
"fields" : [ {
"name" : "subjectivity_id",
"type" : {
"type" : "string",
"logicalType" : "string"
},
"doc" : "Identifier"
}, {
"name" : "topic",
"type" : "string",
"doc" : "Topic name"
}, {
"name" : "created_at",
"type" : [ "null", {
"type" : "long",
"logicalType" : "timestamp-millis"
} ],
"doc" : "Date indicating when the record was created",
"default" : null
}, {
"name" : "updated_at",
"type" : [ "null", {
"type" : "long",
"logicalType" : "timestamp-millis"
} ],
"doc" : "Date indicating when the record was updated",
"default" : null
}]
}
```
```json
{ "subjectivity_id": "subj-1234","topic": "subjectivity_updated","created_at": null,"updated_at": { "long": 1696613296 }}
```
Let me know what other data we can provide!
Contributor guide
Assessment
This issue has not been assessed yet.