confluentinc / confluentinc/confluent-kafka-python

Custom Protobuf Options / Schema Registry Support

Open
#1,470 4 comments 3 reactions 1 assignee Claimed by @pranavrth View on GitHub
component:schema-registry investigate further workaround
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Description
===========
The confluent schema registry (java) recently added support for handling of custom protobuf options. What that meant was that both declarations of custom options, as well as invocations of them, would now get picked up by the registry, whereas before they were just ignored and filtered out:

```
// Declaration
extend .google.protobuf.MessageOptions {
bool my_custom_option = 50000;
}

// Invocation
message MyMessage {
option (my_custom_option) = true;
}
```

However, `confluent-kafka-python` has not kept up with this change. When producing a message using the `ProtobufSerializer` ([here](https://github.com/confluentinc/confluent-kafka-python/blob/master/src/confluent_kafka/schema_registry/protobuf.py#L312)), it uses the `_schema_to_str` method which looks up a schema id from the registry by sending a base64 encoded `FileDescriptor` to match it. This method of matching a schema via the schema registry client's [lookup_schema](https://github.com/confluentinc/confluent-kafka-python/blob/master/src/confluent_kafka/schema_registry/schema_registry_client.py#L384) method works with _declarations_ of custom options, but does not include _invocations_ of them, per above.

That means, that if there are protobuf schemas in the registry with custom options invocations, the python producer will not be able to match those, and will throw a 404 error, and not be able to get a schema id for the produce call.

In the confluent java client, this is not an issue, I believe it's handled [here](https://github.com/confluentinc/schema-registry/blob/master/protobuf-provider/src/main/java/io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema.java#L707-L714).

Are there plans to fix this in `confluent-kafka-python` (and/or the python protoc plugin, not sure where the actual issue is)?

Are there any known workarounds to get this to work prior to such a fix?

Thanks - let me know if I misunderstood something or if I'm looking at this the wrong way.

How to reproduce
================
1. Create a protobuf schema with a custom option in it.
2. Register the schema with the registry as a plain string.
3. The schema registry will now have a correct representation of the schema with a custom option in it.
4. Produce a message using that schema using the `ProtobufSerializer`, setting `auto.register.schemas` to false
5. Producing of the message will fail with a "schema not found" error from the registry.

Checklist
=========

- `confluent-kafka-python` version: 1.9.2
- Running against confluent cloud 7.3

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.