Java SDK : How to bounded messages from PubSub ?
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I want to set up a dataflow Pipeline in Streaming mode.
My dataflow do theses tasks :
* Read messages from pubsub
* Build my "Document" object
* Insert this Document into BigQuery
* Store the initial message into Google Cloud Storage
The code successfully build and run, but it takes a lot of time for some messages.
I think it takes a lot of time because it treats pusub message one by one :
-\> I build a PCollection with messages from pubsub.
```
PCollection rawtickets = pipeline
.apply("Read PubSub Events", PubsubIO.readStrings().fromTopic(TOPIC))
.apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5)))
.triggering(Repeatedly.forever(
AfterPane.elementCountAtLeast(1000))
).withAllowedLateness(Duration.standardSeconds(300))
.discardingFiredPanes()
)
.apply("Make Document ", ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext
c) throws Exception {
String rawXMl = c.element();
/** code for build my object "Document" **/
Documents docTest = new Documents();
docTest.init(rawXMl);
c.output(docTest);
}
}
))
.setCoder(AvroCoder.of(Documents.class));
```
Here a picture of my complete process :
[Complete Process ](https://zupimages.net/up/19/14/w4df.png)
We can see the latency of the process after reading messages.
Concretely, I search to create a Pcollection of N elements. I have tried methods founds here : [https://beam.apache.org/documentation/programming-guide/#triggers](https://beam.apache.org/documentation/programming-guide/#triggers) but it doesn't group my pubsub messages.
How I can batch this process ?
EDIT : I think I must use "GroupByKey" after a window, but that return an error :
```
PCollection rawtickets = pipeline
.apply("Read PubSub Events", PubsubIO.readStrings().fromTopic(TOPIC))
.apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5)))
.triggering(Repeatedly.forever(
AfterPane.elementCountAtLeast(1000))
).withAllowedLateness(Duration.standardSeconds(300))
.discardingFiredPanes()
)
.apply("Make Document ", ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext c) throws Exception
{
String rawXMl = c.element();
/** code for build my object "Document" **/
Documents docTest
= new Documents();
docTest.init(rawXMl);
c.output(docTest);
}
}
))
.setCoder(AvroCoder.of(Documents.class))
.apply("GroupByKey",GroupByKey.create());
```
Error : Wrong 2nd argument type. Found: 'org.apache.beam.sdk.transforms.GroupByKey', required: 'org.apache.beam.sdk.transforms.PTransform,OutputT\>' less... Inspection info: apply (String, org.apache.beam.sdk.transforms.PTransform,OutputT\>) in PCollection cannot be applied to (String, org.apache.beam.sdk.transforms.GroupByKey)
Imported from Jira [BEAM-6970](https://issues.apache.org/jira/browse/BEAM-6970). Original Jira may contain additional context.
Reported by: chloethonin.
Contributor guide
Research direction
Start with the Java pipeline shown in the issue, especially PubsubIO.readStrings(), the Window configuration, and the GroupByKey attempt; compare those transforms with Beam's programming guide on triggers and grouping. Confirm whether the requested bounded batching is supported for this streaming pipeline and document the correct input shape and completion criteria, with a reproducible example or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100