Create.of(...) does not work as expected when spark-runner is configured to streaming mode
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
When the Spark runner is set to streaming mode, and the input `PCollection` is created via `Create.of(...)`, things do not work as expected.
This happens due to the fact the `EVALUATORS` map in `StreamingTransformTranslator` does not have a matching `TransformEvaluator` for the `Create.Values` transform.
I believe this can be fixed by employing a similar approach to the one used for translating `CreateStream`. In fact, it might be quite possible to move things around so that the translation of `Create.of(..)` uses `CreateStream.of(...)` under the hood.
Imported from Jira [BEAM-2464](https://issues.apache.org/jira/browse/BEAM-2464). Original Jira may contain additional context.
Reported by: staslev.
Contributor guide
Research direction
Start by reading the EVALUATORS map and the existing CreateStream translation in StreamingTransformTranslator. Reproduce the issue with a streaming Spark runner and a PCollection created by Create.of(...), then compare the Create.Values path with CreateStream. Done means Create.of(...) translates and runs correctly in streaming mode, with coverage for the regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100