apache / apache/beam

[Feature Request]: IO connector for Salesforce Pub/Sub API

Open
#22,573 0 comments 0 reactions 0 assignees View on GitHub
ideas io new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Consider building a connector for the Salesforce Pub/Sub API.
Salesforce is ubiquitous and such a connector would add value to the Beam community.

Salesforce has a number of API's (batch and 'event driven'), but has recently introduced a modern API based on gRPC and branded as "Pub/Sub API".
Documentation is [here](https://developer.salesforce.com/docs/platform/pub-sub-api/overview) and blog [here](https://medium.com/salesforce-architects/announcing-pub-sub-api-generally-available-3980c9eaf0b7).
There is some samples available [here](https://resources.docs.salesforce.com/rel1/doc/en-us/static/pdf/Salesforce_Pub_Sub_API_Pilot.pdf).
I think a source is most relevant but the same API can be used as sink.
Not clear to me if it would be possible to parallelise the source.

Such a connector would allow a number of architectures, for example when operational Salesforce data needs to be integrated in real-time with a data & AI platform.

I posted this based on Beam Summit 2022 video from Ken in which he mentions the ambition to grow the number of connectors.

Note that I previously also made a feature request to add a similar connector to Google Datastream ([here](https://issuetracker.google.com/issues/234475025)). However Beam would be a lot more flexible and allow for a lot more specific scenarios.

### Issue Priority

Priority: 2

### Issue Component

Component: io-ideas

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.