confluentinc / confluentinc/schema-registry
ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
- Dominant language
- Java
- Stars
- 2.5k
- Forks
- 1.2k
- Avg merge
- 23h 27m
- Merged PRs (30d)
- 65
Description
Hi,
we see this in the Schema-Registry logs, after some hours after be started the service seems to be "stopped"
Log enty:
[2022-11-08 20:10:56,188] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-key, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource)
[2022-11-08 20:10:56,191] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-key/versions HTTP/1.1" 200 12 6 (io.confluent.rest-utils.requests)
[2022-11-08 20:10:56,192] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-value, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource)
[2022-11-08 20:10:56,194] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-value/versions HTTP/1.1" 200 12 3 (io.confluent.rest-utils.requests)
[2022-11-08 20:11:19,089] INFO Stopped NetworkTrafficServerConnector@3b94d659{HTTP/1.1, (http/1.1)}{hrxkfpdc01.hrx.erp:8081} (org.eclipse.jetty.server.AbstractConnector)
[2022-11-08 20:11:19,090] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session)
[2022-11-08 20:11:19,092] INFO Stopped o.e.j.s.ServletContextHandler@4ed5eb72{/ws,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler)
[2022-11-08 20:11:19,100] INFO Stopped o.e.j.s.ServletContextHandler@12f9af83{/,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler)
[2022-11-08 20:11:19,103] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2022-11-08 20:11:19,103] INFO [kafka-store-reader-thread-_schemas]: Shutting down (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,104] INFO [kafka-store-reader-thread-_schemas]: Stopped (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,108] INFO [kafka-store-reader-thread-_schemas]: Shutdown completed (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,110] INFO KafkaStoreReaderThread shutdown complete. (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2022-11-08 20:11:19,112] INFO Kafka store producer shut down (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2022-11-08 20:11:19,112] INFO Kafka store shut down complete (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2022-11-08 20:11:19,114] ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)
org.apache.kafka.common.errors.WakeupException
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:514)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:278)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:164)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:257)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.ensureCoordinatorReady(SchemaRegistryCoordinator.java:223)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.poll(SchemaRegistryCoordinator.java:108)
at io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector$1.run(KafkaGroupLeaderElector.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[2022-11-08 20:11:20,025] INFO SchemaRegistryConfig values:
access.control.allow.headers =
access.control.allow.methods =
access.control.allow.origin =
access.control.skip.options = true
authentication.method = NONE
authentication.realm =
authentication.roles = [*]
authentication.skip.paths = []
avro.compatibility.level = FULL
compression.enable = true
csrf.prevention.enable = false
csrf.prevention.token.endpoint = /csrf
csrf.prevention.token.expiration.minutes = 30
csrf.prevention.token.max.entries = 10000
debug = false
host.name = hrxkfpdc01.hrx.erp
idle.timeout.ms = 30000
inter.instance.headers.whitelist = []
inter.instance.protocol = http
kafkastore.bootstrap.servers = [PLAINTEXT://hrxkfpdc01.hrx.erp:9092]
kafkastore.checkpoint.dir = /tmp
kafkastore.checkpoint.version = 0
kafkastore.connection.url =
kafkastore.group.id =
kafkastore.init.timeout.ms = 60000
kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafkastore.sasl.kerberos.min.time.before.relogin = 60000
kafkastore.sasl.kerberos.service.name =
kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
kafkastore.sasl.mechanism = GSSAPI
kafkastore.security.protocol = PLAINTEXT
kafkastore.ssl.cipher.suites =
kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
kafkastore.ssl.endpoint.identification.algorithm =
kafkastore.ssl.key.password = [hidden]
kafkastore.ssl.keymanager.algorithm = SunX509
kafkastore.ssl.keystore.location =
kafkastore.ssl.keystore.password = [hidden]
kafkastore.ssl.keystore.type = JKS
kafkastore.ssl.protocol = TLS
kafkastore.ssl.provider =
kafkastore.ssl.trustmanager.algorithm = PKIX
kafkastore.ssl.truststore.location =
kafkastore.ssl.truststore.password = [hidden]
kafkastore.ssl.truststore.type = JKS
What can be the root cause ?
Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the shutdown sequence in the log, then inspect KafkaGroupLeaderElector.run, SchemaRegistryCoordinator.poll, and SchemaRegistryCoordinator.ensureCoordinatorReady from the stack trace. Determine what initiates the schema registry shutdown and whether the WakeupException is a consequence or the cause. Done means documenting a supported root cause and the evidence or reproduction needed to distinguish it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100