Transforms could give hints to runners on resource requirements
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
As discussed in BEAM-673 runners can allocate workers to accomplish their work in a better way if they can take into account some hints from the transforms, e.g. a source can hint data locality and with this information the runner can allocate the workers in an better way, this can also be the case with a particular transform that can suggest to the runner to be executed in a worker with a specific resource, e.g. GPU.
Imported from Jira [BEAM-2085](https://issues.apache.org/jira/browse/BEAM-2085). Original Jira may contain additional context.
Reported by: iemejia.
Contributor guide
Research direction
Start by reading the linked BEAM-673 discussion and the original Jira issue BEAM-2085 for the missing context around runner and transform capabilities. Define the agreed scope for hints about data locality and required resources such as GPUs, then confirm how the runner should consume those hints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100