apache / apache/pinot

[Pinot]API to reset a stream to earliest/latest offset

Open
#6,555 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

In scenarios where the topic is corrupted or recreated or when migrating to a new cluster, pinot streaming consumption is affected because of how pinot manages offsets. It would be nice to have an API where we can reset the stream to start from the earliest/latest offset based on the configuration.

Druid provides a similar API to reset a supervisor, which does the offset and streaming tasks management, which makes these situations easier to handle. Reference - https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Pinot's streaming consumption and offset-management entry points; no specific files or tests are identified in the issue. Compare the requested behavior with Druid's supervisor reset API, then define an API that resets a stream to its configured earliest or latest offset and verify the resulting consumption behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.