apache / apache/pinot

More accurate consumption metrics in presence of server restarts and replication

Open
#11,441 4 comments 0 reactions 0 assignees View on GitHub
metrics
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, metrics like `REALTIME_ROWS_CONSUMED` are emitted by each partition consumer, which we can use to track real-time consumption at a topic-level by aggregating the metric per time window. However, the metric is emitted by each replica making it complicated to aggregate---ideally we would just divide the metric by the replication factor, but when replicas restart (and start re-consuming a segment) or fall-behind in consumption, this calculation is no longer accurate.

I'm wondering if there's any way to emit/calculate such a metric, accurately, at the topic-level without too much overhead.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how REALTIME_ROWS_CONSUMED is emitted by partition consumers and replicas, then examine how time-window aggregation could account for restarts, re-consumption, and lag. Done means a topic-level consumption metric remains accurate across those conditions without excessive overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.