apache / apache/beam

Apache Beam Python: Dataframe Transforms break when the option runtime_type_check is enabled.

Open
#21,428 0 comments 0 reactions 1 assignee Claimed by @TheNeuralBit View on GitHub
bug dataframe P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

We have discovered a potential bug whereas when you execute a pipeline that contains
a DataframeTransform with the "runtime_type_check" option set to True, a cryptic
error is raised by Apache Beam typecheckng.

Simple example to reproduce the bug:
    
```

from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam import Pipeline,
Create, Row
from apache_beam.dataframe.transforms import DataframeTransform
pipeline = Pipeline(options=PipelineOptions(runtime_type_check=True))
pipeline
| Create([Row(val1=1)]) | DataframeTransform(lambda df: df)
pipeline.run()
```

This raises a apache_beam.typehints.decorators.TypeCheckError:
```

File ".....lib/python3.8/site-packages/apache_beam/typehints/typehints.py", line 416, in check_constraint
 
  raise SimpleTypeHintError
apache_beam.typehints.decorators.TypeCheckError: According to type-hint
expected output should be of type .
Instead, received 'BeamSchema_118086df_671f_4643_a929_ba65de48e7e8(val1=1)', an instance of type . [while running 'DataframeTransform/Unbatch
'placeholder_DataFrame_140623617251840'/ParDo(_UnbatchNoIndex)']
```

 

Imported from Jira [BEAM-13905](https://issues.apache.org/jira/browse/BEAM-13905). Original Jira may contain additional context.
Reported by: benwah.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.