apache / apache/beam

PubsubIO.readStrings() and DataflowRunner

Open
#19,679 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow direct gcp io java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Hi all,

I have a potential issue I am debugging with `org.apache.beam.sdk.io.gcp.pubsub`. When using the GCP PubSub service no messages are received on the DirectRunner or the DataflowRunner when using PubsubIO.readStrings(). When using the DirectRunner and the Pubsub Emulator it works correctly locally.

I have been working with this issue for the past few days and believe it may be related to the PubsubIO library.

 

I am listening to a topic (also tried a subscription) using:
```

PubsubIO.Read read = PubsubIO
.readStrings()
.withTimestampAttribute("timestamp")

.fromTopic(options.getTopic());
```

 

Alternatively this method works:
```

PubsubIO.Read read = PubsubIO
.readMessagesWithAttributes()
.withTimestampAttribute("timestamp")

.fromTopic(options.getTopic()); 
```

The code works perfectly with the Pubsub Emulator using readStrings() and after switching to readMessagesWithAttributes() everything works correctly on both the emulator as well as GCP.

 

I am using Beam "2.13.0" with "beam-runners-google-cloud-dataflow-java" with Java 8. 

My "gcloud" versions:

Google Cloud SDK 256.0.0
beta 2019.05.17
bigtable
bq 2.0.46
core 2019.07.26
gsutil 4.41
pubsub-emulator 2019.04.26 

Imported from Jira [BEAM-7851](https://issues.apache.org/jira/browse/BEAM-7851). Original Jira may contain additional context.
Reported by: gregorskii.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.