apache / apache/hudi

kafka delay metric not reporting when count drops to zero

Open Beginner friendly
#17,974 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

What happened:

The kafkaDelayCount metric stops being emitted when Kafka lag drops to zero because the metric call in KafkaOffsetGen.getNextOffsetRanges() is inside a checkpoint validation conditional block.

What you expected:

The metric should always be emitted, reporting 0 when caught up rather than disappearing/flatlining from monitoring dashboards.

Steps to reproduce:
Start DeltaStreamer with Kafka source and metrics enabled
Wait for consumer to catch up (latestOffsets == committedOffsets)
Observe kafkaDelayCount metric is no longer emitted

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with KafkaOffsetGen.getNextOffsetRanges(), where the issue says the kafkaDelayCount metric call is inside checkpoint validation. Reproduce the DeltaStreamer scenario with Kafka metrics enabled and the consumer caught up, then verify that the metric remains emitted with a value of 0 when latestOffsets equals committedOffsets.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.