apache / apache/druid

KafkaIndexSupervisor becomes UNHEALTHY, blocking partition consumption

Open
#18,779 1 comment 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

## Druid Version
24.0.1

## Supervisor Type
KafkaIndexSupervisor

## Problem Description
We observed that when a Kafka ingestion task's duration exceeds the configured `segmentGranularity`, the Supervisor enters the `UNHEALTHY_SUPERVISOR` state. As a result, certain Kafka partitions stop being consumed, even if the partition contains a very small amount of data (only a few thousand rows).

This occurs even under minimal load, which indicates that the issue is likely related to Supervisor's internal task handling rather than Kafka volume or resource exhaustion.

Additionally, the logs sometimes show repeated exceptions like:
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value

Contributor guide

Open the contributing guide

Research direction

Start with KafkaIndexSupervisor task handling and the logs containing Optional.get() on an absent value. Reproduce the case where ingestion exceeds segmentGranularity, then trace why the supervisor becomes UNHEALTHY_SUPERVISOR and stops consuming partitions. Done means the supervisor remains healthy and affected partitions continue consuming under the reported low-load conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.