apache / apache/beam

InteractiveRunner cannot execute pipeline with cross-language transform

Open
#20,526 0 comments 0 reactions 0 assignees View on GitHub
bug cross-language P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The InteractiveRunner crashes when given a pipeline that includes a cross-language transform.

Here's the example I tried to run in a jupyter notebook:
```

p = beam.Pipeline(InteractiveRunner())
pc = (p | SqlTransform("""SELECT
CAST(1 AS INT)
AS `id`,
CAST('foo' AS VARCHAR) AS `str`,
CAST(3.14 AS DOUBLE) AS `flt`"""))
df
= interactive_beam.collect(pc)

```

The problem occurs when [pipeline_fragment.py](https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L66) creates a copy of the pipeline by [writing it to proto and reading it back](https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L120). Reading it back fails because some of the pipeline is not written in Python.

Imported from Jira [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). Original Jira may contain additional context.
Reported by: bhulette.

Contributor guide

Open the contributing guide

Research direction

Start with the cross-language pipeline example in a Jupyter notebook, then inspect sdks/python/apache_beam/runners/interactive/pipeline_fragment.py at the pipeline copy logic around lines 66 and 120. Reproduce the failure with SqlTransform and trace the proto write/read path; done means InteractiveRunner can execute the pipeline without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, sql
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.