apache / apache/pulsar-client-go

[bug] Go and Java Schema Incompatibility in Pulsar

Open
#1,031 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

### Description

The issue highlights the incompatibility between Go and Java schemas in Pulsar, specifically focusing on Avro, JSON, Proto, and ProtoNative schemas.

#### Avro Schema

Avro schema is compatible between Go and Java. Example code snippets are provided for both languages to demonstrate the compatibility.

```go
schema := NewAvroSchema(`{"fields":
[
{"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}
],
"name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
```

```java
@AllArgsConstructor
@NoArgsConstructor
public static class Example {
public String name;
public int id;
}

Producer producer = pulsarClient.newProducer(Schema.AVRO(Example.class))
.topic(topic).create();
```

#### JSON Schema
JSON schema requires the field names to be identical and the handling of null fields to be consistent. However, there is a difference in handling null fields between Go and Java. The Java `Schema.JSON(Example.class)` allows null fields implicitly, while Go JSON schema does not permit null fields. To achieve compatibility, the Java example should use a schema definition that allows null fields, and the variable names in the Java `Example` class should match the schema definition.

```go
exampleSchemaDefCompatible := "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":[\"null\", \"string\"]}]}"

consumerJSCompatible := NewJSONSchema(exampleSchemaDefCompatible, nil)

exampleSchemaDefIncompatible := "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"

consumerJSIncompatible := NewJSONSchema(exampleSchemaDefIncompatible, nil)
```

To achieve compatibility, modify the `exampleSchemaDefIncompatible` to allow null fields and ensure that the variable names in the Java Example class match the schema definition.

#### Proto Schema
The Proto schema generated from the same proto message in Go and Java results in different schema definitions that are not compatible. If the Java-generated schema definition is used uniformly, the consumer in Go can register successfully and receive messages. However, the message decoding will still fail. No error is reported, but all the messages only have the default values.
```proto
message TestMessage {
string stringField = 1;
int32 intField = 2;
}
```

Java producer code
```java
for (int i = 0; i < 10; i++) {
producer.newMessage().value(new org.apache.pulsar.client.api.schema.proto.Test.TestMessage().toBuilder()
.setStringField("message").setIntField(i).build()).send();
}
```

Go consumer code
```go
for true {
msg, err := consumer.Receive(context.Background())
assert.Nil(t, err)
err = msg.GetSchemaValue(&unobj)
assert.Nil(t, err)
log.Printf("Receive message %s, %d", unobj.StringField, unobj.IntField)
}
```
Log in Go
```log
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
2023/06/20 20:00:40 Receive message , 0
```

#### ProtoNative Schema
Similar to the Proto schema, ProtoNative schema also generates two incompatible schemas for the same proto message. Using the schema definition from the Java producer to create a Go consumer schema can resolve compatibility issues. However, even with this approach, the consumer will still decode messages with default values.

Please note that the provided examples and code snippets are for illustrative purposes only and may need to be adapted to suit your specific use case.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the cross-language cases using the shown NewAvroSchema, NewJSONSchema, GetSchemaValue, Schema.JSON, and generated Proto message entry points. Compare the Go and Java schema definitions and decoding results for Avro, JSON, Proto, and ProtoNative; done means compatible schemas decode the transmitted field values rather than only default values.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.