KafkaIndexSupervisor becomes UNHEALTHY, blocking partition consumption
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
## Druid Version
24.0.1
## Supervisor Type
KafkaIndexSupervisor
## Problem Description
We observed that when a Kafka ingestion task's duration exceeds the configured `segmentGranularity`, the Supervisor enters the `UNHEALTHY_SUPERVISOR` state. As a result, certain Kafka partitions stop being consumed, even if the partition contains a very small amount of data (only a few thousand rows).
This occurs even under minimal load, which indicates that the issue is likely related to Supervisor's internal task handling rather than Kafka volume or resource exhaustion.
Additionally, the logs sometimes show repeated exceptions like:
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value
Contributor guide
Research direction
Start with KafkaIndexSupervisor task handling and the logs containing Optional.get() on an absent value. Reproduce the case where ingestion exceeds segmentGranularity, then trace why the supervisor becomes UNHEALTHY_SUPERVISOR and stops consuming partitions. Done means the supervisor remains healthy and affected partitions continue consuming under the reported low-load conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100