apache / apache/beam

PubSubIO readMessagesWithMessageId() breaks the payload encoding when using DataflowRunner

Open
#20,246 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow gcp io java P3 pubsub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

My pipeline reads PubSub messages and parses their payload to objects using Gson. I use PubsubIO.readMessagesWithMessageId() to get the PubSub message and the message ID.

I tested the pipeline thoroughly by running it with the DirectRunner in my local machine and everything works fine, but when running it as a Dataflow job in GCP using the DataflowRunner, Gson can't parse the messages properly because the first character of the payload (opening bracket "{") is missing, this only happens when using the DataflowRunner. 

I noticed that the problem no longer happens when using PubsubIO.readStrings() instead of PubsubIO.readMessagesWithMessageId() and getting the payload directly (previously I had to decode the payload using new String(element.getPayload(), StandardCharsets.UTF_8); )

 

 

Imported from Jira [BEAM-10113](https://issues.apache.org/jira/browse/BEAM-10113). Original Jira may contain additional context.
Reported by: alexandermalyga.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.