apache / apache/beam

AfterProcessingTime trigger doesn't create any file panes

Open
#18,574 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Scenario:
I want to count how many events A and B I am getting for given 30 window. I require that every pane has all events types (A and B) with corresponding counters - this is why I am using Combine.globally.

The calculation logic works fine the problem is with writing files. The files are not written.
For debugging purposes I created some transformations (Simulate ApplyShardLabel, Simulate GroupIntoShards etc) that mimics that logic implemented by WriteFiles.

If you push string "A" and "B" to kinesis stream I am seeing the following system.out from the job:

```

AFTER COMBINE: {A=1, B=1}

```

According to my test transformations I should also see:

```

AFTER COMBINE: {A=1, B=1}
Simulating ApplyShardLabel
Simulating finalizing writer: KV{null, [KV{0,
[{A=1, B=1}]}]}

```

Using DirectRunner and Beam 2.0.0. When I switch to Beam 2.1.0 I see the expected debug output and files being written out.

I think that there is some issue with AfterSynchronizedProcessingTime trigger support.

I cannot replicate the issue when using `TestStream`

The test code can be found at
[https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa](https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa)

Imported from Jira [BEAM-3152](https://issues.apache.org/jira/browse/BEAM-3152). Original Jira may contain additional context.
Reported by: pawelbartoszek.

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction in the linked gist and compare Beam 2.0.0 and 2.1.0 behavior around AfterSynchronizedProcessingTime, Combine.globally, and WriteFiles under DirectRunner. Check whether the trigger support differs from the TestStream case; done means the expected pane output appears and files are written for the Kinesis scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.