Proper compatibility check for PROTOBUF_NATIVE schema
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
The current implementation checks only for changing a root message name.
https://github.com/apache/pulsar/blob/34c18704ce759922ce45820321af44b382a28e10/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java#L67-L72
It's definitely not enough. For example, these schemas won't be compatible due to the changed field type:
```proto
message Person {
int32 age = 1;
string name = 2;
}
```
```proto
message Person {
int64 age = 1;
string name = 2;
```
Of course, it's the simplest case. There are many more of them.
### Solution
- Display in Pulsar docs that schema-compatibility-related policies don't work for `PROTOBUF_NATIVE` schema type. 👎
- Implement it in Java on your own. 👍
Probably take [buf.build](https://buf.build/) rules is a good point to start. https://github.com/bufbuild/buf/blob/8b53bf721f3e21eedc3aef3eb0730fdf56d38064/private/bufpkg/bufcheck/bufbreaking/internal/bufbreakingcheck/bufbreakingcheck.go#L338
### Alternatives
- Wait for a miracle 🪄 from Protobuf maintainers.
- Wait for a contribution from Apache Pulsar contributors.
### Anything else?
I understand why it has been implemented this way. By quick googling, I didn't find any ready-to-use JVM-based Protobuf compatibility checkers.
In comparison, Avro has the `org.apache.avro.SchemaValidator` package.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/ProtobufNativeSchemaCompatibilityCheck.java, especially the compatibility logic linked in the issue. Compare the required behavior with the referenced Buf breaking-check rules, then define coverage for changed protobuf field types and the broader compatibility cases; done means PROTOBUF_NATIVE policies no longer accept incompatible schemas or the limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100