AfterProcessingTime trigger doesn't create any file panes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
Scenario:
I want to count how many events A and B I am getting for given 30 window. I require that every pane has all events types (A and B) with corresponding counters - this is why I am using Combine.globally.
The calculation logic works fine the problem is with writing files. The files are not written.
For debugging purposes I created some transformations (Simulate ApplyShardLabel, Simulate GroupIntoShards etc) that mimics that logic implemented by WriteFiles.
If you push string "A" and "B" to kinesis stream I am seeing the following system.out from the job:
```
AFTER COMBINE: {A=1, B=1}
```
According to my test transformations I should also see:
```
AFTER COMBINE: {A=1, B=1}
Simulating ApplyShardLabel
Simulating finalizing writer: KV{null, [KV{0,
[{A=1, B=1}]}]}
```
Using DirectRunner and Beam 2.0.0. When I switch to Beam 2.1.0 I see the expected debug output and files being written out.
I think that there is some issue with AfterSynchronizedProcessingTime trigger support.
I cannot replicate the issue when using `TestStream`
The test code can be found at
[https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa](https://gist.github.com/pbartoszek/9dd58c4fcfc5171eafba3520cb3040fa)
Imported from Jira [BEAM-3152](https://issues.apache.org/jira/browse/BEAM-3152). Original Jira may contain additional context.
Reported by: pawelbartoszek.
Contributor guide
Research direction
Start with the reproduction in the linked gist and compare Beam 2.0.0 and 2.1.0 behavior around AfterSynchronizedProcessingTime, Combine.globally, and WriteFiles under DirectRunner. Check whether the trigger support differs from the TestStream case; done means the expected pane output appears and files are written for the Kinesis scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100