apache / apache/beam

Java SDK : How to bounded messages from PubSub ?

Open
#19,485 0 comments 0 reactions 0 assignees View on GitHub
bug core dataflow java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I want to set up a dataflow Pipeline in Streaming mode.

My dataflow do theses tasks :
* Read messages from pubsub
* Build my "Document" object
* Insert this Document into BigQuery
* Store the initial message into Google Cloud Storage

The code successfully build and run, but it takes a lot of time for some messages.

I think it takes a lot of time because it treats pusub message one by one :

 -\>  I build a PCollection with messages from pubsub.
```

PCollection rawtickets = pipeline
.apply("Read PubSub Events", PubsubIO.readStrings().fromTopic(TOPIC))

.apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5)))

.triggering(Repeatedly.forever(
AfterPane.elementCountAtLeast(1000))
).withAllowedLateness(Duration.standardSeconds(300))

.discardingFiredPanes()
)
.apply("Make Document ", ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext
c) throws Exception {
String rawXMl = c.element();

/** code for build my object "Document" **/

Documents docTest = new Documents();
docTest.init(rawXMl);

c.output(docTest);
}

}
))
.setCoder(AvroCoder.of(Documents.class));

```

 

Here a picture of my complete process :

[Complete Process ](https://zupimages.net/up/19/14/w4df.png)

We can see the latency of the process after reading messages.

Concretely, I search to create a Pcollection of N elements. I have tried methods founds here : [https://beam.apache.org/documentation/programming-guide/#triggers](https://beam.apache.org/documentation/programming-guide/#triggers) but it doesn't group  my pubsub messages.

How I can batch this process  ?

 

EDIT : I think I must use "GroupByKey"  after a window, but that return an error :
```

PCollection rawtickets = pipeline
.apply("Read PubSub Events", PubsubIO.readStrings().fromTopic(TOPIC))

.apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5)))

.triggering(Repeatedly.forever(
AfterPane.elementCountAtLeast(1000))

).withAllowedLateness(Duration.standardSeconds(300))
.discardingFiredPanes()

)
.apply("Make Document ", ParDo.of(new DoFn() {

@ProcessElement
public void processElement(ProcessContext c) throws Exception
{
String rawXMl = c.element();

/** code for build my object "Document" **/
Documents docTest
= new Documents();
docTest.init(rawXMl);


c.output(docTest);
}

}
))
.setCoder(AvroCoder.of(Documents.class))
.apply("GroupByKey",GroupByKey.create());

```

Error : Wrong 2nd argument type. Found: 'org.apache.beam.sdk.transforms.GroupByKey', required: 'org.apache.beam.sdk.transforms.PTransform,OutputT\>' less... Inspection info: apply (String, org.apache.beam.sdk.transforms.PTransform,OutputT\>) in PCollection cannot be applied to (String, org.apache.beam.sdk.transforms.GroupByKey)  

 

 

Imported from Jira [BEAM-6970](https://issues.apache.org/jira/browse/BEAM-6970). Original Jira may contain additional context.
Reported by: chloethonin.

Contributor guide

Open the contributing guide

Research direction

Start with the Java pipeline shown in the issue, especially PubsubIO.readStrings(), the Window configuration, and the GroupByKey attempt; compare those transforms with Beam's programming guide on triggers and grouping. Confirm whether the requested bounded batching is supported for this streaming pipeline and document the correct input shape and completion criteria, with a reproducible example or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.