confluentinc / confluentinc/common

Incompatibility of the plugin with kafka-connect

Open
#737 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17
Forks
243
Avg merge
1d 21h
Merged PRs (30d)
38

Description

Hey, everybody!

I have this situation:

I was using image `confluentinc/cp-kafka-connect:7.7.0` in conjunction with clickhouse-kafka-connect v.1.2.0 and everything worked fine.

After a certain period of time I updated image `confluentinc/cp-kafka-connect` to version 7.7.1. And everything stopped working, an error appeared:
```

java.lang.VerifyError: Bad return type
Exception Details:
Location:
io/confluent/protobuf/MetaProto$Meta.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
Reason:
Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
Current Frame:
bci: @24
flags: { }
locals: { 'io/confluent/protobuf/MetaProto$Meta', integer }
stack: { 'com/google/protobuf/MapField' }
Bytecode:
0000000: 1bab 0010 0001 0018 0100 0001 0000 0002
0000010: 0300 2013 2ab7 0002 b1bb 000f 59bb 1110
0000020: 59b7 0011 1212 b601 131b b660 14b6 0015
0000030: b702 11bf
Stackmap Table:
same_frame(@20)
same_frame(@25)

at io.confluent.protobuf.MetaProto.(MetaProto.java:1112)
at io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema.(ProtobufSchema.java:246)
at io.confluent.kafka.schemaregistry.protobuf.ProtobufSchemaProvider.parseSchemaOrElseThrow(ProtobufSchemaProvider.java:38)
at io.confluent.kafka.schemaregistry.SchemaProvider.parseSchema(SchemaProvider.java:75)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.parseSchema(CachedSchemaRegistryClient.java:301)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:347)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaBySubjectAndId(CachedSchemaRegistryClient.java:472)
at io.confluent.kafka.serializers.protobuf.AbstractKafkaProtobufDeserializer.deserialize(AbstractKafkaProtobufDeserializer.java:138)
at io.confluent.kafka.serializers.protobuf.AbstractKafkaProtobufDeserializer.deserializeWithSchemaAndVersion(AbstractKafkaProtobufDeserializer.java:294)
at io.confluent.connect.protobuf.ProtobufConverter$Deserializer.deserialize(ProtobufConverter.java:200)
at io.confluent.connect.protobuf.ProtobufConverter.toConnectData(ProtobufConverter.java:132)
```

I have this setting in docker-copmpose:

```
kafka-connect:
image: 'confluentinc/cp-kafka-connect:7.7.1'
container_name: connect
hostname: connect
ports:
- '8083:8083'
environment:
CONNECT_BOOTSTRAP_SERVERS: 'kafka:9092'
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-7.6.0.jar
CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components,/usr/share/dockershare"
CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR,com.clickhouse=DEBUG

```
I put the clickhouse-kafka-connect plugin in the `/usr/share/confluent-hub-components` folder. I don't add anything else. All other packages like `com.google.java-protobuf` etc. - come by default in the image (at least I see jar files with those names). I can also run the clickhouse-kafka-connect connector itself and it runs on version 7.7.0 and 7.7.1.

But when I post a message to a topic and kafka-connect starts pulling the message and apparently deserializing it, an error occurs.

By the way, I have the same settings as for `confluentinc/cp-kafka-connect:7.7.0`. But as soon as I change the image version from 7.7.0 to 7.7.1 - the error occurs.

A little searching for a solution - there was a suggestion that it is connected with some incompatibility of package versions, but I can't say for sure.
Can you tell me if someone has encountered this problem and knows how to solve it?
Or maybe someone has some ideas what can be tried to solve the problem.
I will be very grateful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with confluentinc/cp-kafka-connect:7.7.0 and 7.7.1, the clickhouse-kafka-connect plugin, and the provided Docker Compose settings. Start at ProtobufConverter.toConnectData in the stack trace and compare the bundled protobuf and Schema Registry dependencies; done means identifying the incompatible package or version and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.