apache / apache/beam

PubsubIO can't provide PubsubClientFactory

Open
#20,675 0 comments 1 reaction 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

PubsubIO provides a method to pass the PubsubClientFactory.
```

/**
* The default client to write to Pub/Sub is the {@link PubsubJsonClient}, created by the {@link

* PubsubJsonClient.PubsubJsonClientFactory}. This function allows to change the Pub/Sub client
* by
providing another {@link PubsubClient.PubsubClientFactory} like the {@link
* PubsubGrpcClientFactory}.

*/
public Read withClientFactory(PubsubClient.PubsubClientFactory factory) {
return toBuilder().setPubsubClientFactory(factory).build();
}

```

The comment here explains that can pass a PubsubClient.

However, the IncomingMessage used in the {color:#ff0000}pull(){color} method of PubsubClient's abstract method is actually an abstract internal class, in other words, the abstract class PubsubClient can't be extended in external of the {color:#ff0000}org.apache.beam.sdk.io.gcp.pubsub{color} package.

To overcome the above problem, I created the org.apache.beam.sdk.io.gcp.pubsub directory to extend PubsubClient and passes it to the {color:#ff0000}PubsubIO.withClientFactory{color} method.

This seems to run locally without any problems, but when I submit my job to Dataflow, the withClientFactory method is useless and be replaced with the default PubsubJsonClientFactory.

Imported from Jira [BEAM-10936](https://issues.apache.org/jira/browse/BEAM-10936). Original Jira may contain additional context.
Reported by: j.zheng.

Contributor guide

Open the contributing guide

Research direction

Start by tracing PubsubIO.withClientFactory through PubsubClient.pull and the IncomingMessage visibility described in the issue. Compare the local behavior with job submission to Dataflow, focusing on why the supplied factory is replaced by PubsubJsonClientFactory. Done means an externally implemented factory remains effective when running on Dataflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.