confluentinc / confluentinc/ksql

Unable to import topic as stream in Confluent Cloud when using proto schema with referenced message types

Open
#9,733 1 comment 1 reaction 0 assignees View on GitHub
bug protobuf schema-registry-integration
Dominant language
Java
Stars
315
Forks
1k
Avg merge
1d 7h
Merged PRs (30d)
36

Description

**Describe the bug**
I am unable to import a kafka topic as a stream to ksqlDB using Confluent Cloud, when the topic has a proto schema which uses other messages as types.
Seems related to [#8933](https://github.com/confluentinc/ksql/pull/8933) and [#9369](https://github.com/confluentinc/ksql/pull/9369)

**To Reproduce**
Steps to reproduce the behavior, include:
1. KSQL version: 0.28.2 (used from Confluent Cloud)
2. Proto schema of the topic:
```
syntax = "proto3";

option java_multiple_files = true;
option java_package = "com.mycomp.samples";
option java_outer_classname = "SampleRecordProtoDefinition";

message SampleRecord {
int32 my_field1 = 1;
double my_field2 = 2;
string my_field3 = 3;
ReferencedRecord my_field4 = 4;
}
message ReferencedRecord {
string my_field1 = 1;
int32 my_field2 = 2;
}
```
3. Import topic using `Import topics as streams` button
When using the `Import topics as streams` button from ksqlDB I get this error:
```
Error code: 40001. Value schema has multiple schema definitions. - SampleRecord - ReferencedRecord Please specify a schema full name in the WITH clause using VALUE_SCHEMA_FULL_NAME
```
![import_topic_error](https://user-images.githubusercontent.com/52197771/205648572-fa95e799-b9da-4ed2-8449-510f333c3954.png)

4. Import topic using editor and specified `value_schema_full_name` as described in [#8933](https://github.com/confluentinc/ksql/pull/8933)
When importing using the editor with below query:
```
CREATE STREAM PROTO_TOPIC WITH (
kafka_topic = 'proto_topic',
value_format = 'PROTOBUF',
value_schema_full_name = 'com.mycomp.samples.SampleRecord'
);
```

I get this error:

```
Unable to verify if the value schema for topic: proto_topic is compatible with ksqlDB. Reason: null Please see https://github.com/confluentinc/ksql/issues/ to see if this particular reason is already known. If not, please log a new issue, including this full error message. Schema:syntax = "proto3"; option java_multiple_files = true; option java_package = "com.mycomp.samples"; option java_outer_classname = "SampleRecordProtoDefinition"; message SampleRecord { int32 my_field1 = 1; double my_field2 = 2; string my_field3 = 3; ReferencedRecord my_field4 = 4; } message ReferencedRecord { string my_field1 = 1; int32 my_field2 = 2; }
```
![import_topic_error_in_editor](https://user-images.githubusercontent.com/52197771/205649395-8fa83d25-2710-45f7-a6df-d8eb6e429907.png)

**Expected behavior**
I would expect the topic to be imported as a stream and inherit the proto schema.

**Actual behaviour**
I get the error messages specified above.

**Additional context**
When the proto schema uses only scalar types, the topic is imported successfully. The problem seems to be the fact that other message types are referenced.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Confluent Cloud import flow with the provided PROTOBUF schema containing SampleRecord and ReferencedRecord, first through the “Import topics as streams” button and then with the shown CREATE STREAM query. Trace the PROTOBUF schema validation and import path; done means the topic imports as a stream and inherits the referenced-message schema without either reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, sql
Domain
databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.