FlinkRunner: Implement an Artifact service using the Flink DistributedCache
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
We need to have a DistributedCache-based artifact service to ship with the portable Flink runner. The DistributedCache is a perfect fit for Flink because it comes for free and is the mechanism that Flink already uses to distribute its own artifacts.
The final artifact service implementation should be pluggable, but using the DistributedCache allows the Flink runner to work without additional external dependencies (beyond perhaps Docker).
Imported from Jira [BEAM-3672](https://issues.apache.org/jira/browse/BEAM-3672). Original Jira may contain additional context.
Reported by: bsidhom.
Contributor guide
Research direction
Start by reading this issue and the linked BEAM-3672 Jira item, then inspect the portable Flink runner's existing artifact-service entry points and Flink's DistributedCache documentation. Done means a pluggable DistributedCache-based artifact service is integrated so the runner can distribute artifacts without additional external dependencies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100