confluentinc / confluentinc/schema-registry

Schema Registry is continue working even if KafkaStoreReaderThread has been stopped unexpectedly

Open
#3,192 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.5k
Forks
1.2k
Avg merge
23h 27m
Merged PRs (30d)
65

Description

Hello,

I would raise a question about decision behind keeping Schema Registry healthy when KafkaStoreReaderThread died.

We have a case when Kafka Access Control List(ACLs) was applied and KafkaStoreReaderThread has been stopped due to exception [KafkaStoreReader thread has died for an unknown reason](https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/storage/KafkaStoreReaderThread.java#L301)

After that Schema Registry may work only in readonly mode with limited state which has been read before the thread crash.

As I understand, only Schema Registry restart may launch the KafkaStoreReaderThread again. If it's true then how to know that the KafkaStoreReaderThread has been stopped to restart the service(k8s pod) again?

We have an health check for _/subjects_ endpoint, I also tried [/metadata](https://github.com/confluentinc/schema-registry/issues/1267#issuecomment-820389123) endpoint and [k8s tcp health check](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-tcp-liveness-probe) on Schema Registry port.
_But Schema Registry still returns OK response:)_

To summarize:
Do we really need keep Schema Registry working if KafkaStoreReaderThread died? Or kill the Schema Registry itself to let system know about the failure?

As a suggestion, maybe it's better to kill Schema Registry by throwing FatalExitError [here](https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/storage/KafkaStoreReaderThread.java#L301)?

What do you think?

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with core/src/main/java/io/confluent/kafka/schemaregistry/storage/KafkaStoreReaderThread.java at the failure handling around line 301, then review how /subjects, /metadata, and the Kubernetes TCP health check report service health. Done means an agreed behavior for a stopped reader thread and a health signal or shutdown path that exposes the failure to Kubernetes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
api, backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.