confluentinc / confluentinc/schema-registry

ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)

Open
#2,425 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
2.5k
Forks
1.2k
Avg merge
23h 27m
Merged PRs (30d)
65

Description

Hi,
we see this in the Schema-Registry logs, after some hours after be started the service seems to be "stopped"

Log enty:
[2022-11-08 20:10:56,188] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-key, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource)
[2022-11-08 20:10:56,191] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-key/versions HTTP/1.1" 200 12 6 (io.confluent.rest-utils.requests)
[2022-11-08 20:10:56,192] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-value, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource)
[2022-11-08 20:10:56,194] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-value/versions HTTP/1.1" 200 12 3 (io.confluent.rest-utils.requests)
[2022-11-08 20:11:19,089] INFO Stopped NetworkTrafficServerConnector@3b94d659{HTTP/1.1, (http/1.1)}{hrxkfpdc01.hrx.erp:8081} (org.eclipse.jetty.server.AbstractConnector)
[2022-11-08 20:11:19,090] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session)
[2022-11-08 20:11:19,092] INFO Stopped o.e.j.s.ServletContextHandler@4ed5eb72{/ws,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler)
[2022-11-08 20:11:19,100] INFO Stopped o.e.j.s.ServletContextHandler@12f9af83{/,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler)
[2022-11-08 20:11:19,103] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2022-11-08 20:11:19,103] INFO [kafka-store-reader-thread-_schemas]: Shutting down (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,104] INFO [kafka-store-reader-thread-_schemas]: Stopped (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,108] INFO [kafka-store-reader-thread-_schemas]: Shutdown completed (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,110] INFO KafkaStoreReaderThread shutdown complete. (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,112] INFO Kafka store producer shut down (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2022-11-08 20:11:19,112] INFO Kafka store shut down complete (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2022-11-08 20:11:19,114] ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
org.apache.kafka.common.errors.WakeupException
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:514)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:278)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:164)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:257)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.ensureCoordinatorReady(SchemaRegistryCoordinator.java:223)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.poll(SchemaRegistryCoordinator.java:108)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector$1.run(KafkaGroupLeaderElector.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[2022-11-08 20:11:20,025] INFO SchemaRegistryConfig values:
access.control.allow.headers =
access.control.allow.methods =
access.control.allow.origin =
access.control.skip.options = true
authentication.method = NONE
authentication.realm =
authentication.roles = [*]
authentication.skip.paths = []
avro.compatibility.level = FULL
compression.enable = true
csrf.prevention.enable = false
csrf.prevention.token.endpoint = /csrf
csrf.prevention.token.expiration.minutes = 30
csrf.prevention.token.max.entries = 10000
debug = false
host.name = hrxkfpdc01.hrx.erp
idle.timeout.ms = 30000
inter.instance.headers.whitelist = []
inter.instance.protocol = http
kafkastore.bootstrap.servers = [PLAINTEXT://hrxkfpdc01.hrx.erp:9092]
kafkastore.checkpoint.dir = /tmp
kafkastore.checkpoint.version = 0
kafkastore.connection.url =
kafkastore.group.id =
kafkastore.init.timeout.ms = 60000
kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafkastore.sasl.kerberos.min.time.before.relogin = 60000
kafkastore.sasl.kerberos.service.name =
kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
kafkastore.sasl.mechanism = GSSAPI
kafkastore.security.protocol = PLAINTEXT
kafkastore.ssl.cipher.suites =
kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
kafkastore.ssl.endpoint.identification.algorithm =
kafkastore.ssl.key.password = [hidden]
kafkastore.ssl.keymanager.algorithm = SunX509
kafkastore.ssl.keystore.location =
kafkastore.ssl.keystore.password = [hidden]
kafkastore.ssl.keystore.type = JKS
kafkastore.ssl.protocol = TLS
kafkastore.ssl.provider =
kafkastore.ssl.trustmanager.algorithm = PKIX
kafkastore.ssl.truststore.location =
kafkastore.ssl.truststore.password = [hidden]
kafkastore.ssl.truststore.type = JKS

What can be the root cause ?

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the shutdown sequence in the log, then inspect KafkaGroupLeaderElector.run, SchemaRegistryCoordinator.poll, and SchemaRegistryCoordinator.ensureCoordinatorReady from the stack trace. Determine what initiates the schema registry shutdown and whether the WakeupException is a consequence or the cause. Done means documenting a supported root cause and the evidence or reproduction needed to distinguish it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.