apache / apache/beam

[Bug]: Dataflow Harness hangs after MemoryMonitor attempts to shutdown the VM

Open
#22,515 1 comment 0 reactions 1 assignee Claimed by @scwhittle View on GitHub
bug dataflow P2 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Logs on a worker show:

Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 21801/23927/24549 MB, GC last/max = 99.96/165.74 %, #pushbacks=647, gc thrashing=true. Heap dump not written.

After logging this the MemoryMonitor class attempts to shutdown the VM with System.exit(1).
In this case the VM process remained running for a long period of time.

https://stackoverflow.com/questions/2614774/what-can-cause-java-to-keep-running-after-system-exit suggests that this could be due to finalizers. Those might not necessarily be in beam but in loaded libraries due to the pipeline. It suggests a different mechanism for unclean shutdown which I think we should investigate using.

### Issue Priority

Priority: 2

### Issue Component

Component: runner-dataflow

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.