[Bug]: Dataflow Harness hangs after MemoryMonitor attempts to shutdown the VM
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Logs on a worker show:
Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 21801/23927/24549 MB, GC last/max = 99.96/165.74 %, #pushbacks=647, gc thrashing=true. Heap dump not written.
After logging this the MemoryMonitor class attempts to shutdown the VM with System.exit(1).
In this case the VM process remained running for a long period of time.
https://stackoverflow.com/questions/2614774/what-can-cause-java-to-keep-running-after-system-exit suggests that this could be due to finalizers. Those might not necessarily be in beam but in loaded libraries due to the pipeline. It suggests a different mechanism for unclean shutdown which I think we should investigate using.
### Issue Priority
Priority: 2
### Issue Component
Component: runner-dataflow
Contributor guide
Assessment
This issue has not been assessed yet.