apache / apache/pinot

feature: support GCP's PubSub as realtime ingestion

Open
#6,556 3 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

GCP's pubsub implemention doesn't have any kind of partition/offset system so current LLC implementation obviously can't work with it. If people are interested [here is an implementation](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v2.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub) that essentially "fake" offsets and always pull latest messages. However if for some reason the server is stopped while a segment is consuming, the plugin will not re-ingest events that were lost

We also tried to implement it as a HLC ([implementation is here](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v1.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub)) but its pretty much unusable since segments created in HL mode aren't saved to deep storage so you must rely on replicas created through pinot (not handled in the implementation we've made but could be by creating multiple subs).

Since both approach are obviously not doable in production, we have decided to move on to a different pubsub system (most likely apache pulsar) so i don't actually request it to be implemented, i'm pretty much opening it so people looking for this will not waste weeks of research :)

More discussion on slack:
- https://apache-pinot.slack.com/archives/C011C9JHN7R/p1612801563229500
- https://apache-pinot.slack.com/archives/C011C9JHN7R/p1611322225004300

Contributor guide

Open the contributing guide

Research direction

Start with the linked v2.0.0 and v1.0.0 implementations under src/main/java/com/reelevant/pinot/plugins/stream/pubsub, then review the two linked Slack discussions. The issue documents production limitations of both approaches and explicitly does not request implementation, so no concrete completion criteria are defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.