DirectRunner not producing output on TextIO withWindowedWrites() and withNumShards(1)
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Source of bug (Slack user [https://the-asf.slack.com/team/UCVN8DK7V](https://the-asf.slack.com/team/UCVN8DK7V)) and [https://stackoverflow.com/questions/52445414/apache-beam-not-saving-unbounded-data-to-text-file](https://stackoverflow.com/questions/52445414/apache-beam-not-saving-unbounded-data-to-text-file).
Example provided:
```
public static void main(String[] args) {
ExerciseOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(ExerciseOptions.class);
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply("Read Messages from Pubsub",
PubsubIO
.readStrings()
.fromTopic(options.getTopicName()))
.apply("Set
event timestamp", ParDo.of(new DoFn() {
@ProcessElement
public void
processElement(ProcessContext context) {
context.outputWithTimestamp(context.element(), Instant.now());
}
}))
.apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5))))
.apply("Write to File",
TextIO
.write()
.withWindowedWrites()
.withNumShards(1)
.to(options.getOutputPrefix()));
pipeline.run();
}
```
Produces output when executed on the DataflowRunner, does not produce output on the DirectRunner.
Imported from Jira [BEAM-5488](https://issues.apache.org/jira/browse/BEAM-5488). Original Jira may contain additional context.
Reported by: lcwik.
Contributor guide
Assessment
This issue has not been assessed yet.