apache / apache/pulsar

Proper compatibility check for PROTOBUF_NATIVE schema

Open
#19,385 2 comments 0 reactions 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

The current implementation checks only for changing a root message name.

https://github.com/apache/pulsar/blob/34c18704ce759922ce45820321af44b382a28e10/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java#L67-L72

It's definitely not enough. For example, these schemas won't be compatible due to the changed field type:

```proto
message Person {
int32 age = 1;
string name = 2;
}
```

```proto
message Person {
int64 age = 1;
string name = 2;
```

Of course, it's the simplest case. There are many more of them.

### Solution

- Display in Pulsar docs that schema-compatibility-related policies don't work for `PROTOBUF_NATIVE` schema type. 👎
- Implement it in Java on your own. 👍

Probably take [buf.build](https://buf.build/) rules is a good point to start. https://github.com/bufbuild/buf/blob/8b53bf721f3e21eedc3aef3eb0730fdf56d38064/private/bufpkg/bufcheck/bufbreaking/internal/bufbreakingcheck/bufbreakingcheck.go#L338

### Alternatives

- Wait for a miracle 🪄 from Protobuf maintainers.
- Wait for a contribution from Apache Pulsar contributors.

### Anything else?

I understand why it has been implemented this way. By quick googling, I didn't find any ready-to-use JVM-based Protobuf compatibility checkers.

In comparison, Avro has the `org.apache.avro.SchemaValidator` package.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java, especially the compatibility logic linked in the issue. Compare the required behavior with the referenced Buf breaking-check rules, then define coverage for changed protobuf field types and the broader compatibility cases; done means PROTOBUF_NATIVE policies no longer accept incompatible schemas or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.