GoogleCloudPlatform / GoogleCloudPlatform/training-data-analyst
tfdv_basic_spending.ipynb: version of pyarrow installed incompatible with current version of Apache Beam; breaks generate_statistics_from_dataframe
- Dominant language
- Jupyter Notebook
- Stars
- 8.6k
- Forks
- 6.1k
- Avg merge
- 4h 44m
- Merged PRs (30d)
- 2
Description
On the notebook:
training-data-analyst/courses/machine_learning/deepdive2/production_ml/labs/tfdv_basic_spending.ipynb,
The block where we install tfdv and its dependencies, this installs a version of pyarrow that appears is may not be supported in the current version of Apache Beam (in the GCP environment / context).
This breaks tfdv.generate_statistics_from_dataframe(any_pd_df), as it will return a value error with any dataframe. Following the stack trace, I see that the root cause appears that it may be the error "missing package pyarrow.vendored". This line may be the issue:
`!pip install pyarrow==2.0.0`
It looks like by default, version 5.0.0 is installed (as of time of this writing). This line rolls back version 5.0.0 to 2.0.0. In any case, to troubleshoot the ValueError, when I went back, inserted a new cell, and executed the line below, then I retried tfdv.generate_statistics_from_dataframe(...) ... the value error did not appear any longer, and the issue was apparently fixed:
```
!pip install --upgrade pyarrow
import pyarrow
```
**Recommended fix / patch:**
**replace the line**
`!pip install pyarrow==2.0.0`
**with**
```
!pip install pyarrow==5.0.0
# or other suitable version that doesn't break
# the current version of beam
```
Contributor guide
Assessment
This issue has not been assessed yet.