[Pinot]API to reset a stream to earliest/latest offset
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
In scenarios where the topic is corrupted or recreated or when migrating to a new cluster, pinot streaming consumption is affected because of how pinot manages offsets. It would be nice to have an API where we can reset the stream to start from the earliest/latest offset based on the configuration.
Druid provides a similar API to reset a supervisor, which does the offset and streaming tasks management, which makes these situations easier to handle. Reference - https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors
Contributor guide
Research direction
Start by reviewing Pinot's streaming consumption and offset-management entry points; no specific files or tests are identified in the issue. Compare the requested behavior with Druid's supervisor reset API, then define an API that resets a stream to its configured earliest or latest offset and verify the resulting consumption behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100