apache / apache/beam

Tear down unused DoFns periodically in Java SDK harness

Open
#19,907 0 comments 0 reactions 0 assignees View on GitHub
harness improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Per the discussion in the ML the detail can be found here[1], the teardown of DoFns should be supported in the portability framework. It happens at two places:
1) Upon the control service termination
2) Tear down the unused DoFns periodically

The aim of this JIRA is to add support for tear down the unused DoFns periodically in Java SDK harness.

[1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E

Imported from Jira [BEAM-8620](https://issues.apache.org/jira/browse/BEAM-8620). Original Jira may contain additional context.
Reported by: sunjincheng121.

Contributor guide

Open the contributing guide

Research direction

Start with the Java SDK harness and the portability framework discussion linked in the issue. Determine how unused DoFns are identified and where periodic teardown belongs, then verify that unused DoFns are torn down without affecting active processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.