Debezium Vitess connector error [DBZ-8192]

Migrated from [DBZ-8192](https://issues.redhat.com/browse/DBZ-8192)

h2. What Debezium connector do you use and what version?

Debezium vitess connector 2.7.1 Final (also tried with 3.0.0.Beta1)
h2. What is the connector configuration?
{code:java}
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: debezium-connector-betonred-cx
labels:
strimzi.io/cluster: debezium-connect-cluster
spec:
class: io.debezium.connector.vitess.VitessConnector
tasksMax: 1
autoRestart:
enabled: true
maxRestarts: 2
state: running
config:
errors.max.retries: 2
schema.history.internal.consumer.sasl.mechanism: SCRAM-SHA-512
schema.history.internal.consumer.security.protocol: SASL_PLAINTEXT
schema.history.internal.producer.sasl.mechanism: SCRAM-SHA-512
schema.history.internal.producer.security.protocol: SASL_PLAINTEXT
max.request.size: 52000000
producer.max.request.size: 52000000
database.hostname: *****.net
database.port: 15992
database.server.id: 155120
database.connectionTimeZone: Europe/Moscow
vitess.keyspace: betonred_cx
vitess.tablet.type: REPLICA
vitess.vtctld.host: vitess103.marfa-tech.net
vitess.vtctld.port: 15999
# topics settings
topic.prefix: betonred_cx
table.exclude.list: betonred_cx\.\S*[^A-Za-z0-9]\S*
schema.history.internal.kafka.bootstrap.servers: kafka-cluster-kafka-bootstrap:9092
schema.history.internal.kafka.topic: schema-changes.betonred_cx
provide.transaction.metadata: true
key.converter: io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url: http://schema-registry:8081
value.converter: io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url: http://schema-registry:8081
snapshot.mode: never
tombstones.on.delete: false
{code}
h2. What is the captured database version and mode of deployment?

Vitess 17.06
h2. What behavior do you expect?

Correct work of connector
h2. What behavior do you see?

Connector failed with errors
h2. Do you see the same behaviour using the latest released Debezium version?

Yes
h2. Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)
{code:java}
2024-08-30 12:03:38,751 ERROR [debezium-connector-betonred-cx|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [debezium-vitessconnector-betonred_cx-change-event-source-coordinator]
java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
  at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)
  at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:101)
  at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)
  at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.newChannel(VitessReplicationConnection.java:338)
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.startStreaming(VitessReplicationConnection.java:83)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:67)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
{code}
{code:java}
2024-08-30 12:03:38,751 ERROR [debezium-connector-betonred-cx|task-0] Failed to close replicationConnection (io.debezium.connector.vitess.VitessStreamingChangeEventSource) [debezium-vitessconnector-betonred_cx-change-event-source-coordinator]
java.lang.NullPointerException: Cannot invoke "io.grpc.ManagedChannel.shutdownNow()" because the return value of "java.util.concurrent.atomic.AtomicReference.get()" is null
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.close(VitessReplicationConnection.java:350)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:84)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
{code}
{code:java}
2024-08-30 12:03:39,250 ERROR [debezium-connector-betonred-cx|task-0] WorkerSourceTask{id=debezium-connector-betonred-cx-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-debezium-connector-betonred-cx-0] org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.   at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:79)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)   at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)   at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)   at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)   at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype   at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)   at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)   at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)   at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)   at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:101)   at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)   at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)   at io.debezium.connector.vitess.connection.VitessReplicationConnection.newChannel(VitessReplicationConnection.java:338)   at io.debezium.connector.vitess.connection.VitessReplicationConnection.startStreaming(VitessReplicationConnection.java:83)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:67)   ... 9 more

{code}
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

Use this Dockerfile for build docker image
{code:java}
FROM alpine as downloader

ENV DEBEZIUM_CONNECTOR_MYSQL_VERSION='2.7.0.Final' \
DEBEZIUM_CONNECTOR_POSTGRES_VERSION='2.7.0.Final' \
DEBEZIUM_CONNECTOR_VITESS_VERSION='2.7.0.Final' \
CONFLUENT_AVRO_VERSION='7.6.2' \
APACHE_AVRO='1.11.3'

RUN mkdir -p /plugins/confluent
WORKDIR /plugins
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/$DEBEZIUM_CONNECTOR_MYSQL_VERSION/debezium-connector-mysql-$DEBEZIUM_CONNECTOR_MYSQL_VERSION-plugin.tar.gz
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/$DEBEZIUM_CONNECTOR_POSTGRES_VERSION/debezium-connector-postgres-$DEBEZIUM_CONNECTOR_POSTGRES_VERSION-plugin.tar.gz
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-vitess/$DEBEZIUM_CONNECTOR_VITESS_VERSION/debezium-connector-vitess-$DEBEZIUM_CONNECTOR_VITESS_VERSION-plugin.tar.gz

RUN wget -P /plugins/confluent https://repo1.maven.org/maven2/org/apache/avro/avro/$APACHE_AVRO/avro-$APACHE_AVRO.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/common-config/$CONFLUENT_AVRO_VERSION/common-config-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/common-utils/$CONFLUENT_AVRO_VERSION/common-utils-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/$CONFLUENT_AVRO_VERSION/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-connect-avro-data/$CONFLUENT_AVRO_VERSION/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/$CONFLUENT_AVRO_VERSION/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-serializer/$CONFLUENT_AVRO_VERSION/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/$CONFLUENT_AVRO_VERSION/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-converter/$CONFLUENT_AVRO_VERSION/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar

RUN for f in *.tar.gz; do tar -zxvf "$f"; done

# Image used by strimzi operator for your kafka cluser
FROM quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
RUN mkdir -p /opt/kafka/plugins/confluent
ENV CONFLUENT_AVRO_VERSION='7.6.2'
ENV APACHE_AVRO='1.11.3'

COPY --from=downloader /plugins/debezium-connector-mysql/* /opt/kafka/plugins/debezium-connector-mysql/
COPY --from=downloader /plugins/debezium-connector-postgres/* /opt/kafka/plugins/debezium-connector-postgres/
COPY --from=downloader /plugins/debezium-connector-vitess/* /opt/kafka/plugins/debezium-connector-vitess/
COPY --from=downloader /plugins/confluent/common-config-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/common-config-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/common-utils-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/common-utils-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/avro-$APACHE_AVRO.jar /opt/kafka/plugins/confluent/avro-$APACHE_AVRO.jar

USER 1001 {code}

Start with VitessReplicationConnection.newChannel at line 338 and close at line 350, then inspect VitessStreamingChangeEventSource and the supplied Dockerfile reproduction. Run the connector with the shown plugin versions and configuration to isolate the gRPC provider loading failure. Done means the Vitess connector starts streaming without the ServiceConfigurationError and the failure path closes cleanly.

Written by the indexing model from the issue text.