feature: support GCP's PubSub as realtime ingestion
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
GCP's pubsub implemention doesn't have any kind of partition/offset system so current LLC implementation obviously can't work with it. If people are interested [here is an implementation](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v2.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub) that essentially "fake" offsets and always pull latest messages. However if for some reason the server is stopped while a segment is consuming, the plugin will not re-ingest events that were lost
We also tried to implement it as a HLC ([implementation is here](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v1.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub)) but its pretty much unusable since segments created in HL mode aren't saved to deep storage so you must rely on replicas created through pinot (not handled in the implementation we've made but could be by creating multiple subs).
Since both approach are obviously not doable in production, we have decided to move on to a different pubsub system (most likely apache pulsar) so i don't actually request it to be implemented, i'm pretty much opening it so people looking for this will not waste weeks of research :)
More discussion on slack:
- https://apache-pinot.slack.com/archives/C011C9JHN7R/p1612801563229500
- https://apache-pinot.slack.com/archives/C011C9JHN7R/p1611322225004300
Contributor guide
Research direction
Start with the linked v2.0.0 and v1.0.0 implementations under src/main/java/com/reelevant/pinot/plugins/stream/pubsub, then review the two linked Slack discussions. The issue documents production limitations of both approaches and explicitly does not request implementation, so no concrete completion criteria are defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100