apache / apache/beam

Allow to change the PubsubClientFactory when using PubsubIO

Open
#19,162 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When we use PubSub to push or pull messages, we currently have no choice of serialization implementation because PubsubIO use internaly the *_PubsubJsonClient.FACTORY_* of type _*PubsubJsonClientFactory*_ to serialize or deserialize messages.

It should be nice to be able to set a different factory (e.g. *_PubsubGrpcClientFactory_*) to decrease the size of messages (and then decrease the price of using Pubsub).

 

I guess It could be possible to write something like:

`PubsubIO.Write write =`
`    PubsubIO.writeMessages()`
`            .to("projects/project/topics/topic")`
`            *.withFactory(PubsubGrpcClient.FACTORY)*`
`            .withTimestampAttribute("timestamp")``;`

or 

`PubsubIO.Read read = `
`    PubsubIO.readMessages()`
`            .fromSubscription("projects/project/subscriptions/name")`
`            *.withFactory(PubsubGrpcClient.FACTORY)*`
`            .withTimestampAttribute("timestamp");`

 

Imported from Jira [BEAM-5676](https://issues.apache.org/jira/browse/BEAM-5676). Original Jira may contain additional context.
Reported by: lhauspie.

Contributor guide

Open the contributing guide

Research direction

Start by reading the PubsubIO readMessages and writeMessages entry points and how they currently use PubsubJsonClientFactory. Compare the mentioned PubsubGrpcClientFactory integration points for both reading and writing; done means callers can select a factory and messages are serialized and deserialized through it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.