Apache Beam Python: Dataframe Transforms break when the option runtime_type_check is enabled.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
We have discovered a potential bug whereas when you execute a pipeline that contains
a DataframeTransform with the "runtime_type_check" option set to True, a cryptic
error is raised by Apache Beam typecheckng.
Simple example to reproduce the bug:
```
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam import Pipeline,
Create, Row
from apache_beam.dataframe.transforms import DataframeTransform
pipeline = Pipeline(options=PipelineOptions(runtime_type_check=True))
pipeline
| Create([Row(val1=1)]) | DataframeTransform(lambda df: df)
pipeline.run()
```
This raises a apache_beam.typehints.decorators.TypeCheckError:
```
File ".....lib/python3.8/site-packages/apache_beam/typehints/typehints.py", line 416, in check_constraint
raise SimpleTypeHintError
apache_beam.typehints.decorators.TypeCheckError: According to type-hint
expected output should be of type .
Instead, received 'BeamSchema_118086df_671f_4643_a929_ba65de48e7e8(val1=1)', an instance of type . [while running 'DataframeTransform/Unbatch
'placeholder_DataFrame_140623617251840'/ParDo(_UnbatchNoIndex)']
```
Imported from Jira [BEAM-13905](https://issues.apache.org/jira/browse/BEAM-13905). Original Jira may contain additional context.
Reported by: benwah.
Contributor guide
Assessment
This issue has not been assessed yet.