apache / apache/beam

Triggers not working for bounded data

Open
#19,326 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

pCollection
.apply(Window.into(FixedWindows.of(Duration.millis(100)))
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1000)))
.discardingFiredPanes().withAllowedLateness(Duration.ZERO))
.apply(TextIO.write().withWindowedWrites().withNumShards(1).to('gs-path'));

Here pCollection is a *bounded* PCollection. I'm trying to break it into files of 1000 roughly, but all I get is 2 files one having 1000 other having the rest of the data.
If instead I do:-

pCollection
.apply(new GlobalWindow())
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1000)))
.discardingFiredPanes().withAllowedLateness(Duration.ZERO))
.apply(TextIO.write().withWindowedWrites().withNumShards(1).to('gs-path'));

I get just one file. 
Both of the above cases should have conceptually divided the records into chucks of 1000 to be written in a file.

Imported from Jira [BEAM-6743](https://issues.apache.org/jira/browse/BEAM-6743). Original Jira may contain additional context.
Reported by: aditya.guru149.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.