apache / apache/beam

Allow using side inputs as ValueProviders

Open
#18,464 1 comment 0 reactions 0 assignees View on GitHub
core java new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Allow dynamically computed information in a pipeline which is materialized as a side input to be used in the same places as ValueProviders.

This could solve the problem mentioned in [https://stackoverflow.com/questions/45649244/google-cloud-dataflow-apache-beam-can-i-use-sideinputs-with-textio-write](https://stackoverflow.com/questions/45649244/google-cloud-dataflow-apache-beam-can-i-use-sideinputs-with-textio-write) if TextSink supported ValueProviders for headers/footers.

Imported from Jira [BEAM-2771](https://issues.apache.org/jira/browse/BEAM-2771). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Apache Beam's ValueProvider and side-input concepts, then inspect the TextSink/TextIO write entry point mentioned in the issue. Use the linked Jira ticket and Stack Overflow discussion for context. Done should mean dynamically materialized side inputs can be used in the same places as ValueProviders, including the cited headers and footers case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.