apache / apache/beam

GroupByKey doesn't seem to work with FixedWindows in DirectRunner

Open
#20,850 1 comment 3 reactions 0 assignees View on GitHub
bug extensions gcp java P3 pubsub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

After applying `FixedWindows` on a streaming source, a `GroupByKey` operation won't emit keyed elements in a window. 

This example without `GroupByKey` prints all the windowed elements:

 
```

pipeline
.apply("ReadFromPubsub", PubsubIO.readStrings().fromSubscription(subscriptionPath))
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(5L))))

.apply(WithKeys.of("bobcat"))
.apply(MapElements.into(TypeDescriptors.nulls()).via(
(KV pair) -> {
LOG.info("Key: " + pair.getKey() + "\tValue: " + pair.getValue());

return null;
}
));
```

 

This example with `GroupByKey` doesn't emit anything:

 
```

pipeline
.apply("ReadFromPubsub", PubsubIO.readStrings().fromSubscription(subscriptionPath))
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(5L))))

.apply(WithKeys.of("bobcat"))
.apply(GroupByKey.create())
.apply(FlatMapElements.into(TypeDescriptors.nulls()).via(

(KV> pair) -> {
pair.getValue().forEach(message -> LOG.info("Message:
" + message));
return null;
}
));
```

 

I'm using DirectRunner. The same logic works for Python using both the DirectRunner and DataflowRunner.

Imported from Jira [BEAM-12075](https://issues.apache.org/jira/browse/BEAM-12075). Original Jira may contain additional context.
Reported by: tianzi.

Contributor guide

Open the contributing guide

Research direction

Run the Java reproducer with DirectRunner, using PubsubIO, FixedWindows, WithKeys, GroupByKey.create(), and FlatMapElements. Trace the windowed grouped output and compare it with the working example without GroupByKey. Done means keyed elements are emitted from each fixed window as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.