InteractiveRunner cannot execute pipeline with cross-language transform
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The InteractiveRunner crashes when given a pipeline that includes a cross-language transform.
Here's the example I tried to run in a jupyter notebook:
```
p = beam.Pipeline(InteractiveRunner())
pc = (p | SqlTransform("""SELECT
CAST(1 AS INT)
AS `id`,
CAST('foo' AS VARCHAR) AS `str`,
CAST(3.14 AS DOUBLE) AS `flt`"""))
df
= interactive_beam.collect(pc)
```
The problem occurs when [pipeline_fragment.py](https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L66) creates a copy of the pipeline by [writing it to proto and reading it back](https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L120). Reading it back fails because some of the pipeline is not written in Python.
Imported from Jira [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). Original Jira may contain additional context.
Reported by: bhulette.
Contributor guide
Research direction
Start with the cross-language pipeline example in a Jupyter notebook, then inspect sdks/python/apache_beam/runners/interactive/pipeline_fragment.py at the pipeline copy logic around lines 66 and 120. Reproduce the failure with SqlTransform and trace the proto write/read path; done means InteractiveRunner can execute the pipeline without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100