apache / apache/beam

Transforms could give hints to runners on resource requirements

Open
#18,370 1 comment 0 reactions 0 assignees View on GitHub
beam-model core improvement java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

As discussed in BEAM-673 runners can allocate workers to accomplish their work in a better way if they can take into account some hints from the transforms, e.g. a source can hint data locality and with this information the runner can allocate the workers in an better way, this can also be the case with a particular transform that can suggest to the runner to be executed in a worker with a specific resource, e.g. GPU.

Imported from Jira [BEAM-2085](https://issues.apache.org/jira/browse/BEAM-2085). Original Jira may contain additional context.
Reported by: iemejia.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked BEAM-673 discussion and the original Jira issue BEAM-2085 for the missing context around runner and transform capabilities. Define the agreed scope for hints about data locality and required resources such as GPUs, then confirm how the runner should consume those hints.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.