apache / apache/beam

Create.of(...) does not work as expected when spark-runner is configured to streaming mode

Open
#18,408 0 comments 0 reactions 0 assignees View on GitHub
bug P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When the Spark runner is set to streaming mode, and the input `PCollection` is created via `Create.of(...)`, things do not work as expected.

This happens due to the fact the `EVALUATORS` map in `StreamingTransformTranslator` does not have a matching `TransformEvaluator` for the `Create.Values` transform.

I believe this can be fixed by employing a similar approach to the one used for translating `CreateStream`. In fact, it might be quite possible to move things around so that the translation of `Create.of(..)` uses `CreateStream.of(...)` under the hood.

Imported from Jira [BEAM-2464](https://issues.apache.org/jira/browse/BEAM-2464). Original Jira may contain additional context.
Reported by: staslev.

Contributor guide

Open the contributing guide

Research direction

Start by reading the EVALUATORS map and the existing CreateStream translation in StreamingTransformTranslator. Reproduce the issue with a streaming Spark runner and a PCollection created by Create.of(...), then compare the Create.Values path with CreateStream. Done means Create.of(...) translates and runs correctly in streaming mode, with coverage for the regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.